Beep boop - this is a robot. A new show has been posted to TWiT…
What are your thoughts about today’s show? We’d love to hear from you!
Another oil those propeller-head bearings when Steve talks about the Intel Ring researchers. Urbana is pronounced click here
I’m glad that Paul responded to Steve’s criticisms on WW.
Exchange is a mission critical system for most companies that deploy it. Therefore it must work. Addressing those 4 zero days requires somebody to track down exactly where the problem is, test it, understand it, come up with a workaround that won’t break anything else, program that change, then run the thousands of unit test cases to ensure nothing broke, then integrate it into the main system and run the 10s of thousands of system tests. Go back and fix whatever broke, rinse and repeat.
Changes on such a system aren’t changing one line of code, pointed to by the reporting entity, recompiling and publishing.
If Microsoft sat on their combined butts for 2 months, then, yes, they deserve a good kicking. If, on the other hand they were working furiously behind the scenes to get the fix out there (and they patched 2010, 2013, 2016 and 2019, so that is a lot of recoding and testing and there were huge paradigm shifts between 2010, 2013 and 2016, so it isn’t necessarily a matter of changing the same lines of code in each version, or at least the surrounding code may have been extensively modified, so the testing of what might break is very different).
As was said on WW, it would have been worse if Microsoft had publicly acknowledged the problem in January and then rushed a release out the door last week. The more people who know about the problem, the more people who can develop attacks. There is also no mitigation for these bugs, so customers would have been 100% vulnderable to attack between January and the release of the patch either way and if it was widely known, they would have been open to even more attacks.
I’m not giving Microsoft a pass here, but working on a huge code base with thousands of developers is very different to working on a small system written by one or 2 programmers. I’ve worked on both types of project over the years. On a small project, with 1 or 2 programmers, you can usually track down a bug in a couple of hours and address it often within a day and have tested it and released a patch within a couple of days. On a large project, your support team will need to plough through millions of lines of code to find the problem. Even if the reporter has “located” the issue at a certain point, the bug can often be somewhere totally different that has been called by a library used by a library, used by a library, used by a library… of the code that “has” the problem. Until you have jumped completely through the stack, you just don’t know. Then, if it is a called piece of code, you need to ensure the change hasn’t broken anything in the thousands of other instances in the code, where it is called.
That makes a huge difference to the perception of how quickly you respond.
Without seeing Microsoft’s timesheets for the last 2 months, it is hard to tell, whether they didn’t care and quickly threw out a patch slung together in the last week or whether they really had whole teams going at this for 2 months.
Given how Exchange updates, which weren’t rushed, often cause massive problems and often need the patch to be rolled back, data restored, configuration changed and re-patched, this isn’t something Microsoft will throw out the door without properly testing it.
2 Likes
While watching I couldn’t help thinking that maybe what happened is that the Solar Winds fiasco allowed information about the Exchange Server vulnerabilities to leak. For example the company that found and reported the problem may have been backdoored and their proof of concept was stolen by attackers and quickly used to backdoor Exchange Servers even while Microsoft was working on the patch and to test it and produce something they could reliably send to the field.
3 Likes
Yes, that ran through my mind as well. Although, as Microsoft was compromised and parts of Exchange source exfiltrated, I thought it might have come from there.
Thanks for this reminder. It is really easy to play Monday morning quarterback (sorry, I don’t know the European equivalent) and say “Gee, obviously this company should have done/not done XYZ”, but unless you’re on the inside you don’t fully appreciate the scope of the problem or all the considerations that must be taken. As a reformed Nokia Lumia fan, I am reminded of the constant consternation about “worthy upgrades” and why Nokia wasn’t just making the perfect phone that would satisfy everyone.
2 Likes
I would hope that companies start to question their use of this product after this. How can people put up with such crappy support for so important a system? It almost seems deliberate to encourage people to move over to the cloud based exchange service which was not impacted by these bugs.
Security is hard, so why do people keep trusting MS given it’s track record?
Armchair General is the term I grew up with.
2 Likes
Because there aren’t many competitors that provide anything near what Exchange can do. There is Lotus Domino, which makes Exchange seem like a magnificent solution, Open Xchange or whatever it is called today, which was not a bad product, once you had invested months and tonnes of money learning how to configure and administer it, it would be cheaper and quicker to use Exchange.
There are some cloud services coming along, which can almost compete, but I’ve yet to find one that can compete head to head.
It’s integration into Office, Sharepoint and ability to build up workflows and share mailboxes and folders makes it very flexible, but it also makes it cumbersome, in security terms.
1 Like
As a completely irrelevant aside on the subject of Monday Morning Quarterbacks / Armchair Generals, it’s my unscientific observation that any comment that starts with “Surely…” is usually a waste of breath/keystrokes.
I believe I’ve mentioned it before, but a team of very very clever people I had the honour to work alongside and learn from had a big poster in their room with the slogan: “For every complex problem, there’s a simple answer - and it’s wrong”.
2 Likes
WinAmp is always one of the first thing I install on any new computer although, I suppose, what is the point? I haven’t bought music for about five years, not even in MP3 form, but instead I use Spotify so I question the wisedom of the WACUP project. If it had been released 10 years ago then I probably would have jumped on it
Microsoft should probably get the benefit of the doubt on the Exchange issue although I’d have thought that most people, except those with regulatory compliance issues, would have moved to Microsoft 365.
It could be that, because Microsoft had to go back to Exchange 2010, there were subtle differences in each version and they wanted to get them all out at the same time and you want all of the myriad number of server, operating system, and hardware configurations, to work properly
On Linux you could have an obsolete WinAmp clone that hasn’t been updated since 2007 in XMMS. If you want something kept up to date you should check out Clementine https://www.clementine-player.org/ , available on multiple platforms.
I think WinAmp is still being offered for download from Ninite
Here comes more pain for Exchange Servers that were previously backdoored:
Post updated to remove “7,000” from the headline and to make clear not all of them have been infected with ransomware.. but still bad and what’s next.