Beep boop - this is a robot. A new show has been posted to TWiT…
What are your thoughts about today’s show? We’d love to hear from you!
1 Like
Hi @Leo and Steve,
In this latest episode, Steve did a great job of describing what happened, and the UMN researcher perspective from their explanation pdfs, and there was some discussion from Leo about what it actually means to be “submitted” and what a “commit” is when working with the Linux kernel.
I think I can help provide some background on how the Linux kernel’s development actually happens.
I have been a kernel developer for 11 years, and I maintain a small subsystem of the kernel on behalf of my company. Being a subsystem maintainer technically means that I have permissions to write to some Linux kernel repositories that eventually flow into live kernel releases, but the way that Linus and the other core maintainers have set things up prevent me and others like me (with limited domain expertise and maintainership scope) from modifying the kernel willy nilly without good justification for each change.
1 Like
One critical factor of this story that I think is missed in the broader tech coverage is exactly how the Linux kernel takes on proposed changes: It’s done entirely via email.
The same for code reviews. There is no slick web code review interface (like Github’s) for the kernel. It’s done with email responses with inline text.
See: Submitting patches: the essential guide to getting your code into the kernel — The Linux Kernel documentation
or: Submitting Your First Patch to the Linux Kernel and Responding to Feedback
It has been known that Linux kernel community are very particular about rules for email formatting. For example, if you send rich-text or html formatted email to any linux list, prepare for it to be completely ignored or filtered. If you top-post in a response, someone on the list will grumble at you. If you use Exchange or Microsoft Outlook, probably your emails sent to the list (including patches) will gum up the works, and won’t be acted on.
The reason why email is treated so seriously (including the actions that the UMN researchers) is that email is entirely in the critical path of the kernel maintainer workflow.
This was part of the Q&A from the UMN researchers about whether their bad changes actually landed in the linux kernel’s git repositories:
We did not introduce or intend to introduce any bug or
vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email
exchanges, without being adopted or merged into any Linux branch, which was explicitly
confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even
become a Git commit in any Linux branch.
It’s important to realize what the UMN researchers did here. They 100% admit to emailing a “v1” of a patch to the relevant maintainer and mailing list that they knowingly introduce a carefully crafted vulnerability. In the parlance of kernel maintainership, that’s a “submission” to the kernel. Based on a close reading of their paper, they would wait until the maintainer would acknowledge their proposed change with an email response, before immediately responding to the maintainer with a, “Wait, nevermind, there’s a mistake. Please take v2 instead.”
Once a maintainer confirmed our patches, e.g., an email
reply indicating “looks good”, we immediately notify the
maintainers of the introduced UAF and request them to not
go ahead to apply the patch. At the same time, we point out
the correct fixing of the bug and provide our correct patch.
In all the three cases, maintainers explicitly acknowledged
and confirmed to not move forward with the incorrect patches.
To me, it does seem like they are playing with fire a little bit here… If the maintainer had accepted and merged the v1 of the patch, and no other action was taken, then within a few months time, a major version of the Linux kernel would be released with the bug live.
It’s not out of the question that an email is lost in transit, or the maintainer wasn’t paying attention to follow up emails, and the bad patch slipped through. The whole point of their paper is that maintainers (and the process) are sometimes not careful.
4 Likes
@Leo asked during the podcast whether the “banning” of the UMN email addresses, and the fact that UMN email addresses had commits in the live kernel meant that the UMN researchers had commit access to the kernel repositories and therefore had some credentials or access revoked.
Nope. That’s not what happened. To my knowledge, no one from UMN had that kind of access to begin with.
Leo was correct that only a select group of people have write access to kernel dot org git repositories. People who have kernel dot org accounts. However, only kernel maintainers and high-profile developers are allowed these accounts and can commit new changes to repositories bound for Linus’s releases.
A non-maintainer has to send their patch (via email) to a maintainer, who signs off on it, and makes a commit in their git tree, but the originator of the work still gets authorship credit in the kernel.
Git has a concept of “author” and “committer” which need not be the same entity.
Just as an example, here’s a commit of mine that landed not long ago, where I’m the author, but not the committer (GregKH, the maintainer at the center of this drama, just happened to be the committer, as this is in one of his trees):
https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git/commit/?h=usb-next&id=29b01295a829fba7399ee84afff4e64660e49f04
author Benson Leung bleung@chromium.org 2021-01-28 22:14:03 -0800
committer Greg Kroah-Hartman gregkh@linuxfoundation.org 2021-02-01 15:31:34 +0100
commit 29b01295a829fba7399ee84afff4e64660e49f04usb: typec: Add typec_partner_set_pd_revision
The partner’s PD revision may be resolved later than the port partner
registration since the port partner creation may take place once
Type-C detects the port has changed state, but before PD communication is
completed.Add a setter so that the partner’s PD revision can be attached to it once
it becomes available.If the revision is set to a valid version (not 0), the setter will also
refresh the partner’s usb_pd flag and notify on “supports_usb_power_delivery”
sysfs property as well.Reviewed-by: Heikki Krogerus heikki.krogerus@linux.intel.com
Signed-off-by: Benson Leung bleung@chromium.org
Link: [PATCH 3/6] usb: typec: Add typec_partner_set_pd_revision - Benson Leung
Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org
When GregKH wrote that changes from UMN are banned, it’s really a directive to his fellow Kernel Maintainers that they shall not merge patches from those email addresses into their maintainer trees, not that any ACL privilege was revoked.
1 Like
What the UMN researchers totally failed to do is follow the best practices of this sort of penetration testing. You have a mandate that is signed off on by the client. Only a set few people know what is going on, nobody else (in this case on the list) is informed and they are tested - but with the full knowledge of the project leaders - and there are strict limits on what is to be done and what they cannot do.
If they had received the go-ahead from Greg Kroah-Hartmann or Linus Torvalds and the procedure was tested with their knowledge (i.e. if they noticed these patches coming in, they would keep quiet until the relevant maintainers had OKed or rejected the submissions), there wouldn’t be this problem.
I agree with Steve, that what they did was important research and was an important test of the Linux submission process and that they also did their best to ensure that the code did not get into production. But the way they went about it was wrong, it broke the principles of white hat pentesting and has ruined the UMN’s reputation in the process.
1 Like
Greg KH did this talk on kernel development, and describes how patches are submitted and flow into a new version of the kernel. Set to a timestamp with hopefully an enlightening flow diagram.
The UMN researchers were acting as one of the green “developers” at the bottom of the flow diagram, and their bad patches tested the file and subsystem maintainers above them.
It’s not until the driver/file maintainer or subsystem maintainer level that the users actually have write access to kernel dot org’s git repositories.
I also want to point out that this diagram isn’t one of a structured pre thought out process between machines or entities. The whole process grew organically and is ultimately based on human-to-human trust. Linus cannot review every patch, every change that comes in, so he trusts people to take care of large and small parts of the kernel.
Linus trusts Greg on USB matters, so Greg is the USB maintainer, and so on. What the UMN research is focused on is the trust of that 1st stage, between a random developer sending a bug fix, and the maintainer. As a maintainer, we are mainly used to people putting in effort to make a proposed change in order to fix a problem on their own system or use case, or to generally improve the kernel. Implicitly, there is (a perhaps unfounded) trust that developers won’t send us a deliberately bad change.
I think this is why the UMN research rubs kernel maintainers the wrong way, because it challenges the notion that people who email us are doing so with the best of intentions.
Perhaps we do need the wakeup call though. Had a truly bad actor done this instead of the researchers, this could have been a serious supply chain attack involving a widely-used open source project.
2 Likes
Precisely. There have been a couple of cases of this in OSS over the years. Everybody should be aware of this possibility and double check everything. I do this with normal emails, let alone something as important as a Kernel code change.
The UMN’s goal was valid and the community should take on their findings and treat it as a warning. But how they went about it was flawed.
Edit: Just realised that my comment about email could be misconstrued. I am not a Kernel maintainer! 
Just when I get emails with the same sort of gravity, I take extra care to ensure they are genuine and don’t contain mistakes.
1 Like
Really great input from people who know, thanks so much for the update and clarification. I’ll make sure Steve sees this, too.
2 Likes
One of the things that occured to me, under the topic of the DoD and AS8003.
Well, if I was a DoD InfoSec researcher, I might wonder “Who out there would just blindly accept DoD advertisements without any kind of verification?” I think the mystery and anonymity in the beginning was part of the point. Which backbones and transit providers will send potentially sensitive and high security routes off to some anonymous clown in Florida.
I also would expect that aspect of it failed, at least apparently, the routes were propagated. From the NANOG thread someone called HE and was told the routes were legit, it sure would be interesting to see some transparency on this, if there was a proper backchannel before the routes were accepted, that would improve my confidence in route security, but I’m afraid it’s not the case,.
Incidentally, I just realized who you are! Welcome, Benson, and thanks for your great work on Type-C cables!!
4 Likes
Thanks, @Leo!
I’m looking forward to helping folks on this community on USB Type-C related topics too.
3 Likes
Hey @Leo and Steve.
The Linux Foundation’s Technical Advisory Board has posted a public report of the UMN incident, if you want the kernel maintainer’s perspective.
The report is very detailed, and goes through patch by patch with a play-by-play of every commit referenced from the UMN paper, and a commentary on each one.
https://lwn.net/ml/linux-kernel/202105051005.49BFABCE@keescook/
Thanks to the open nature of the Linux kernel mailing lists, the TAB report is able to point to open mailing list links to each of the commits only mentioned by the UMN paper so that everyone can see what they did.
Among the things that the TAB report reveals is the fact that UMN’s research used fake identities (George Acosta and James Bond) to submit changes, which is explicitly against the Kernel’s Developer Certificate of Origin rules.
Thanks,
Benson
3 Likes
And Steve was moaning about the length of time it took to patch Exchange…
Exim was told about patches in October last year, patched, with the help of the reporter by the end of February, released in April and a public announcement about the vulnerabilities released yesterday.
I have a lot of respect for Steve, but when it comes to large projects (and comparing Exchange to Chrome is like comparing repairing a Cessna, with repairing a F22), there is a magnitude of difference in complexity of such projects.
Comparing Exim and Exchange would be a fairer comparison - and, in fact, both projects needed similar timelines to get patches out, Exim maybe a bit more. Also, in the Exim case, Qualys were given full access to the Git repository and helped with auditing and then fixing the code.
Very interesting. Thanks for the update Benson. I was sure there was more to the story.