Sobering reading Log4J

TWiT Shows Security Now
1

After 2 years, 1 in 4 apps are still vulnderable.

In a survery, Veracode found that 79% of developers never bothered to update external modules or libraries, once they had been integrated into their projects - which means any external code that has security vulnerabilities and is fixed by the original project won’t filter down to the apps that actually use that code in anger!

A lot of apps that were patched for Log4J are still running the first patch against Log4J, which itself had a major RCE flaw (remote code execution), implying that many developers patched the original Log4J, but fell back on old habits of not checking security bulletins or installing patches when they are released.

Around 35% of apps are still vulnerable to Log4J and 40% vulnerable to the RCE that came during the panic period after Log4J was discovered.

2 Likes
2

I’m not advocating for companies/developers burying their heads in the sand, but on the other hand, I think the phrase “still vulnerable” needs clarification. What these statistics are actually saying is something like “there are many applications still built using a library known to have vulnerabilities.” Just because that library is present, does not mean there exists a path through the application (from user input) that yields a vulnerability. In the particular case of Log4J, there needs to be a particular type of logging operation requested, one of which is not that commonly used by developers. There is also the mitigating factor that the logging configuration can frequently be edited in the field by end users, disabling logs, or removing portions of the logged data that could pose a risk.

So, yes, developers should always keep abreast of the risks out there, BUT perhaps many have been paying attention, and realize the problem is mostly one of news organizations going after clicks… because otherwise there would be examples of compromised apps all over the place and surely that would make the news and generate more clicks.

3

Even so, there is no excuse for sloppy security and version control.

It if you think you program doesn’t use Log4J (or some other library) in a way that it could be exploited, what other libraries are there, that might use it that way?

Why release software you know has severe security risks built in? In any other industry, you’d end up in court for such negligence.

4

That’s precisely my point though, the dev KNOWS if s/he is using the vulnerable portion of Log4J. Very few people ever used the specific feature that was vulnerable, and it’s not like it can accidentally be added to the program if it wasn’t there to start with. Is a vulnerability really a vulnerability if the vulnerable code CANNOT be executed? My argument is this recent round of press is mostly click-bait because they’re calling it a vulnerability, when in all likelihood it’s just dormant code that will never be a risk to anyone. YES, is should still be addressed, if for no other reason than to quiet down these foolish click-bait press events… but the sky is NOT falling.

5

My point is, they know there is a version that isn’t vulnerable at all, so why run the risk?

And, again, if they are using Log4J, they are probably using other libraries that they have never looked at the code, how can they be sure that they aren’t using the vulnderable features? This isn’t specifically about Log4J, just the very lacadazical attitude, if you are using third party libraries or add-ons, you should be ensuring you are using a safe version, if it is known that there is a flaw.

This is something I learnt back in the 80’s and early 90’s. When developing for Windows in the 90s, before the Internet was big, we had a role on projects whose job was to monitor all suppliers of development libraries and controls for security or bug fixes and to keep the team informed and ensure updates were integrated in a timely manner.

When I was doing PHP programming in the late 2000s and early 2010s, it was the same, although I did a couple of 1 man projects as well, so I ensured I was signed up to the developer newsletters for each library or framework I was using and getting critical fixed integrated into my project as quickly as possible.

6

Don’t think I am defending bad practices, just trying to explain why they can occur. As someone who built an entire stack of software in Java as a career, getting all the various parts to work together was difficult. You can have entire chains of dependencies, and trying to replace just one is a lot more work, if it’s even possible, than you would expect. The best “failure” that occurs in this case is a failure to build, at least then you know you have a problem. Instead though, what can happen, is some little hidden thing breaks, and you therefore need to go through a full round of testing, even though you didn’t change one line of your own code. It’s an expense that not every business is willing to endure… and also a risk… so thus bad decisions sometimes get made. Additionally, the conservative business nature “programs” developers who want to make changes to not even try, because they always get shot down due to risk.

The way to force the updates is to make it a bigger financial risk not to… so probably requires some sort of regulation.

2 Likes
7

To amplify what PHolder is saying, appart from the unexpected behaviour when you upgrade a library like this, what might also happen is a cascade of upgrades.
If you upgrade one library then you might end up having to upgrade several other libraries because the new version of the library is incompatible with the older existing version of other libraries it depends on.
The dependency graph and its resolution is quite complicated and most of the time teams tend to avoid rocking the boat.
The best way to avoid this is to try and minimise the change as much as possible, so if for example you have version 1.2.3 which is vulnerable but you know that 1.2.4 is not but is still quite old because version 2.0 exists then you do the patchlevel upgrade and hope for the best.
In addition to all this, there are also contractual issues, for example someone has purchased a product that is using the vulnerable version and they do not have a support contract for patching. They do not have the budget or want to pay more for an upgrade, so they keep using the vulnerable version, until they get hacked.

8

The last one is up to the user, if they don’t want to pay for support, that is their problem.

But knowingly putting customers/users in danger is a different matter. For cosmetic or feature updates, I can understand not always pushing updates, but when it comes to security, there should be no excuses.

9

In this case when I mentioned that someone had purchased a product I was not referring to consumers but rather businesses. They in their turn provide a service to customers that is insecure because of their choice to not upgrade their infrastructure.
Generally i agree with you that people should not ignore security but the reality is that too many organisations do not think about it at all which is why we have ransomware crippling schools, train companies, public sector services, power companies etc.

1 Like
10

We’ve thrown out several suppliers in recent years, because they weren’t supplying security updates or security updates in a timely manner.

There are some older products that just don’t have updates, which can’t be replaced, these get isolated - we have a lot of working lab equipment, where the latest version of the controlling software runs on XP. Those machines are either not networked or they are on a sealed segment, where they can only see each other.

But that is our choice - $750,000 to replace a perfectly working piece of kit, just because the controlling PC has to run XP is a difficult choice. Luckily, the PCs don’t have to be online or generally communicate with each other.

At another employer, I took over the IT manager position. The previous external service provider had set every user’s password to 123456 (so they could provide support) and all of them had email accessible over OWA on the Internet!

My first day was spent locking all Internet access to their accounts and forcing password resets!

So, I do understand that a lot of companies don’t care about security.

I’ve also worked on the other side, writing software for large corporations, and we always ensured that known security updates were pushed out quickly. Likewise, I helped develop several online shops and provided securit updates for them.

I did a security report (before cybersecurity was a word) for a UK retailer who was moving online. I pointed out dozens of security holes in their system. The managers didn’t want to spend money on fixing it and the programmers didn’t like someone external being paid to look at their code and show them where there were problems…

I showed them that anyone could log on, using any users account. Not interested. I showed how I could pull out all the customer details and credit card information. Not really that bothered(!)

Eventually, I got frustrated and using the logon screen, I dropped the whole development database from the server… That opened their eyes and they got busy sorting out the problems…

2 Likes
11

Yeah, that is still the state of things for many companies. That is why lots of vulnerabilities go unpatched for years.