The Impact of Log4j in the Open-Source Ecosystem
Charlie Guarino: Hi everybody. This is Charlie Guarino. Welcome to another edition of TechTalk SMB. I'm sitting here today with the business architect for open-source technology for IBM i Jesse Gorzinski. Jesse, thank you for joining us here today for this podcast.
Jesse Gorzinski: You bet. Thanks for having me on Charlie.
Charlie: Absolutely, thanks. It's so good to talk to you once again Jesse, always a pleasure. Jesse I was reading the other day about problems that we're having in technology and specifically I was reading for Log4j—
Jesse: Oh sure.
Charlie: And somehow in my mind I thought that was behind us. But it seems, based on what I was reading, that we're still potentially suffering the aftereffects of this—and in fact some companies are dealing with it as if it still had just happened today. I was quite surprised to hear that. What are you seeing out there in the landscape?
Jesse: Yeah, that's—I mean that's absolutely true. We saw the headlines, I believe back in December. There was kind of a mass panic back in December, and that's of course when all the publications were talking about the Log4j vulnerability known as Log4Shell or sometimes LogJam, and it really was in terms of you know, people logging on Medium, people out on Twitter. It was a big deal and a lot of IT folks were scrambling to address this big deal and a lot of folks doing a great job, a lot of vendors stepping up and helping to make sure that their software is patched appropriately and so on. But it really is more of a long-standing problem for people because it did expose a few weaknesses in what people were following when it comes to best practices for managing their open-source software or even their proprietary software for that matter, right? And so that's why we're still seeing the aftereffects today is because you know people in general are looking for, you know, ways not only to make sure that they are secure from the current vulnerability but also prepared for the future. Because that's a key point and that's still again why it's getting a lot of focus in some circles is making sure people are prepared for the future, because this is likely to happen again.
Charlie: Jesse for the—for those who don't have a broad understanding of Log4j in general, can you just briefly describe what the problem was or what we were dealing with?
Jesse: Yeah absolutely and it was rated you know the highest score by the CVE Council—I forget what their name is but the reason it was such a high visibility issue and why it got so much press back in December is because it's a vulnerability in a logging framework, and in order to exploit the vulnerability all you needed to do is somehow trigger a malicious string to be written through that logging facility, right? So if you have an application that's sending stuff to log files or publishing events to you know an active MQ, JMS que or something like that, if you somehow get that vulnerable or malicious string logged through that framework, Log4j would actually go out and connect to a remote system to get more information, right? So it used this JNDI technology in Java but it would detect this keyword and it would actually go connect to a remote system and start getting code to execute to get more problem detail or information about whatever event it is logging, which is a very powerful feature but if you're a malicious user, you can exploit that and have it run some malicious code, right? So you can stand up your own malicious code and be able to tell somebody to run it, which is why it was so bad, right, you know because it also then didn't require—it didn't require somebody to be a PhD hacker extraordinaire to be able to exploit necessarily. It's just—it's in a logging framework and so that's why it was so pervasive.
Charlie: And my impression of everything I've read about this and people whom I've spoken with, it seems that we were really caught flatfooted on this. Like this came out—it literally just appeared one day in the news, and then it kind of—it never stopped since then. So that kind of begs of the question how was it that we were seemingly caught flatfooted on this? I mean I guess it was a hack I suppose, right, but that's how these things typically happen.
Jesse: Yeah, and I would say there's two reasons for this. Reason #1 is simply because the target software Log4j is so common. It is used by a vast majority of Java applications that are running in the enterprise, so you're talking web application servers. You're talking message buses, ESBs—you know, REST API back ends. There's so much out there running these Java workloads in the enterprise. We all know Java continues to be one of the most popular languages, has been for a very long time. It has a very large ecosystem especially for enterprise, and so now you have a logging framework which is used by a very large portion of all of these enterprise applications, and so when that—there's a vulnerability, you really have to look at any application that's running Java. And so the number of you know, just the quantity of things that you need to analyze in order to get a handle of how exposed you were, how exposed you are, how to mitigate—there's just a lot to deal with. That's reason #1, right? But you couple that with reason #2, which quantity, is why I chose the topic today of best practices. Reason #2 is that there's probably some lessons to be learned from IT folk, and everything I'm saying today is not specific to IBM i, or even specific to open source for the most part. It is that there's a lot of people when IT or when this Log4j vulnerability broke out, they didn't know for sure what their entire software stack looked like, right? And so I published a blog article back in December, or maybe it was January, where I talked about like some best practices for helping to avoid this stuff, and then #1 is just maintaining a software inventory, right? A lot of people had a software inventory that was deficient or you know, just not kept up to date. So when something like this hit and it's like, well, any Java application in your enterprise can now be affected, right? Maybe you're running Tomcat on a Linux server somewhere. You're running some REST back end or something on a Windows server. All of those things are now possible attack vectors, but there were plenty of companies that said wait, we don't even know what's running, right? So you should be able to look at your system, whatever your system is, and look at for instance all the ports that are listening and be able to confidently say what's running on each of those ports. What are all these services? And so this inventory is a huge part of the best practices, just understanding what software you're running because that's going to help you identify your action plan, [and] also help you identify which vendors you need to work with for any third party software that you're running to make sure that if there is a vulnerability, that it's patched.
Charlie: You know what comes to my mind here Jesse after hearing you say this. I don't—I hate to use the word complacency because that sounds awfully insulting perhaps, but I know some shops when everything is running well—and this is not unique to open source, this is also running any platform. They're not keeping up with updates for example, things like that or they may not even—as you said the stack is old in some cases. It just makes me think doesn't that speak to how important it is to stay current and to keep your systems running optimally using the proper software fixes and patches.
Jesse: Oh yeah absolutely, and you know in some cases that's very apparent, but it is easy to lose track of you know the age of your software that's running whether it's you know again anywhere in your enterprise, whether it's your web front end code or something running on the back end on pick your platform, right? It's very easy to let that stuff fall out of date and it also again falls back on that software inventory. Know what you're running and know to keep it up to date, but then couple that with the due diligence that is required to know when you need to update these things, right? So for instance all the major language runtimes—PHP, Ruby, Python, R, Rust—all of the major programming languages have very good release cycles published up front, right? So for instance if you're running a Node.js application, you can go to the Node.js release working group web page and you can see, okay, the version I'm running is going to reach end of life on such and such date. So you need to have a plan in place to migrate to a newer version and do proper testing in advance of the end of life date. So that keeps you on a version that's always supported and most open—major open-source projects have that, right? Most third party proprietary vendor applications have some form of that, so there really is this due diligence that's required just to stay current with things.
Charlie: But there's always that question of balance, and what I mean by that is balance meaning I hear people say well I don't want to be the pioneer. I don't be the first one in a new package, either. I want it to, you know, go out in the industry and simmer out there for a little bit of a time before I bring it into my own shop. But the other side of that spectrum of course is now you've gone too long—
Jesse: Right. Right.
Charlie: So how do you find that proper balance?
Jesse: Right, and that's a really good question. That probably depends on a lot of factors like what industry you're running, what is your network topology, how many things do you have that are externally visible services, things along those lines. As a general rule, I recommend like once every two months you should analyze your software stack to see if there is anything that needs to be going through an update process, something that needs to be getting its version bumped up, right? You know so—because when we've had security vulnerabilities, whether it was you know Hartley back in the day or Log4Shell or any of the others that have popped up, it's been a lot easier for the people to deal with who are already at a recent version of the software. Because if you have to bump from, you know, a version that's only six months old to the latest version that has all the best fixes, that's usually a very non-disruptive path. But if you say oh wow, the version we're running is eight years old, ten years old and we need to get to a new version now to have something addressed whether it's a security vulnerability or an integrity issue or maintainability issue, whatever your issue with it is, that can be really disruptive because in those ten years it could have gone through several major versions. There could have you know binary incompatibility, source incompatibility, so your programs won't work with the new version, so that's an issue. So if you kind of you know pay the price in small increments, it's a lot easier than ending up having to go through a big, disruptive process in an emergency scenario.
Charlie: Right and this one obviously hit the headlines, but it's only when you have these big, headline-busting vulnerabilities that are exposed that are jarring—you know they're stark reminders of how important it is to stay current. But it brings up another question: So now we—so if I'm going to continue adopting newer to open-source technologies, I sure hope that there is some governance in place perhaps, or some deity or somebody or some group of people—
Jesse: Right.
Charlie: Who are responsible for maintaining the integrity of this code.
Jesse: Right, yeah, and that's actually part of a very important discussion these days, which is you know who owns open-source software, and as a consumer of open-source software, what guarantees do you have, right? Who can you turn to if you need help with anything, right? There's a lot of questions that you need to understand the answers to to properly embrace open source, right? A lot of open source is backed by major companies, right, so all the major pieces of software out there—you know Node.js is backed by Google and IBM and PayPal and you know all kinds of companies out there. You know Walmart has donated, you know some of the stuff that they used to build their front ends to the community. You have OpenSSL, which has you know just millions and millions of dollars’ worth of developer time from a lot of major companies out there, right? And so you have a lot of open-source projects that have the proper resources to get things done and even governance in place to have a structure behind when they do updates and so on. But you do end up falling into some portions of the world—whether it's again open source or proprietary—it's not unique to open source. You might be running packages that only have a couple guys who are maintaining it or one person that's maintaining it, and that is maybe a risk that you'd want to consider as you're analyzing what you run your company on, right? And so there's this whole governance model of you know who is keeping this alive? Is there a software foundation that provides some structure to the lifecycle of these things? So the Apache Software Foundation, the Eclipse Software Foundation—those are good—OpenJS Foundation. Those are really good at helping to ensure solid reliable deliverables as well. You need to do your due diligence when analyzing what software to use—again, whether that's open source or otherwise. Open source has the unique capability—and I call it a best practice—to help give back, right? So even something as small as reporting issues. We've heard of companies who say oh yeah, I tried this open-source software thing and for some reason if I didn't configure it a certain way, it didn't work. Well maybe that's an issue that you need to raise to the maintainers because they might not have known about it, right? Even something as simple as opening issues or helping with documentation or spreading awareness, right? I've heard numerous times people saying oh yeah, this piece of software doesn't have good documentation. Well if it’s open source—I mean if it's proprietary software, go complain. If it's open-source software, yeah sure. Open an issue or give back and help that documentation out, right? That's going to help with the maintainability of the code and you know overall that's going to allow the maintainers to focus on other things, and it's going to in a very indirect way help the overall security and stability of that software.
Charlie: I mentioned how it takes these large events to really to jar us and to really—to refocus back on this topic, but surely there are other things happening all along. I mean what other incidents might there have been recently that might be perhaps equally as jarring, or certainly noteworthy and worthy of discussion, to the proper people in your organization?
Jesse: Yeah, so there was a really kind of amusing one in this package called Colors.js, which is a JavaScript package used by Node.js applications for anything that was writing stuff to a terminal because it provided colorization, right? You wanted warnings to be yellow and errors to be red. They would use this very prevalent package known as Colors.js to just do that colorization. So not a lot of code there and you know there's a lot of question about the motivations for the maintainers doing this, but the maintainers put out a minor fix update to that code that if you went to use it, it essentially printed out an American flag and then went into an infinite loop. So it just pegged your CPU, right? So you had all kinds of applications that were suddenly—if they weren't managed properly—suddenly just printing out an American flag and hanging. You know there are lessons to be had there as well in terms of you know, you can talk about governance and things like that, but you could also just talk about, again, best practices which include knowing the proper toolset for the software stack that you're running right? So and the Colors.js incident for example, you know there's the NPM tool, which is what goes and fetches software from the central repository to install it into your project to run. Everybody knows about NPM install but not enough people know about NPM-CI. NPM-CI is there to ensure that when you run NPM-CI, you can recreate the exact same software stack that you used in tests, right, or used wherever you established your package list. And so if you're spinning new Docker containers or something like that and you weren't running the NPM commands properly, you could inadvertently pull in this new package and it broke something, right? But the people who understood the toolset understood when to use NPM Install vs. NPM-CI. It was not an issue at all, right, because you just didn't have some new version downloaded automatically if you were rebuilding it for you know, a fresh container or something like that.
Charlie: Jesse is it a fair question to—or fair point to make that as sure as you and I are sitting here discussing this very topic, that this will happen again in some other incarnation, somehow. Is that a real statement to make or a fair statement to make?
Jesse: Oh yeah that's a very safe statement to make, right? We all know cyber security continues to be a major concern among IT professionals. It continues to be a major growing source of concern as there are more and more data breaches that happen, more and more IT infrastructures that end up compromised, malware, ransomware. I mean we're just seeing this stuff in a trend that's not going downward right, so yes. There is going to be more things like this, whether it's in open-source software or proprietary software. You know I remember one my first forays into this adventure where I worked for a previous—my previous employer before IBM I worked in the IT department for a company in the finance industry, and we were hit with the Bugbear virus. And that was—I mean this is 20 years ago now, but it was this very prevalent virus that essentially allowed you to, or it would infect, pretty much any internet-connected computer that was a Windows computer, right? Things like this happen and things like this will continue to happen. But the folks who are following the best practices of knowing their software stack, knowing it intimately, knowing what software they're running, what language it is using, what versions they are running, keeping informed of what's going on with that software—so maybe they know about trends and directions before it hits the headlines and also being able to respond quickly. Those people will be able to handle any future events like this confidently, and we saw that with several vulnerabilities that have been exposed in the past is that there are people who handled it very, very well. You can handle it confidently and very well if you keep a good software inventory, you stay up to date, and you keep informed of what's going on.
Charlie: Right, so we're not just sitting ducks. There are proper ways to manage this risk certainly, and how to be best prepared. So you've already mentioned things like due diligence and keeping your software stack current. Are there any other things that come to your mind about how we can be better prepared or prepared as best as we possibly can so that the next time this does in fact happen that we're not going to be caught up in the wrath of this thing?
Jesse: Yeah, so a couple of things. I didn't really talk today about keeping informed, but that's a really important element, right? There are security vulnerabilities for all kind of software—again, open source and proprietary—all the time, and there's databases that talk about security vulnerabilities. So you should make some effort to make sure that you're informed before it hits the big headlines, right? In fact part of the open-source support for you know—I'm assuming you know enough people running open source have support in place from IBM or from somewhere—the IBM open-source support offering includes this OSPRI proactive reporting offering that will proactively tell you about changes in your software stack, whether that's new versions with new features or some security vulnerability that may or may not be actionable of you, and so you can get alerted proactively about some of this stuff happening. And I'm not meaning to be a salesmen for the IBM offering or anything, but do what you can to just stay on top of things, right? If you're running third party software from pick your vendor, make sure that you're on their mailing lists or on whatever contact method they have for letting their customers know if there's an issue, right, because that's really key you know. And so again you couple all those things—you can do all of this very efficiently and very confidently.
Charlie: And this is—and this discussion far exceeds just a security breach per se. There are other things that—
Jesse: Yeah.
Charlie: Can affected by this thing. I mean think about like just maintaining the code, things like that. What are your thoughts on something like that?
Jesse: Yeah, exactly right. This—you know following some simple best practices helps with you know you mentioned it: maintainability, auditability, right? Imagine if an auditor comes in and says what software are you running on this server over here, and you say I don't know. There's a bunch of stuff, right? I can run a PS on my Linux system or a WRKACTJOB on my IBM i system and there's a whole bunch of stuff there. If you can't confidently say what everything is, that's—that has an impact on your auditability position, right? How's your posture when somebody comes in and tries to analyze what's going on, right?
Charlie: Is it prudent then, Jesse, to have somebody come in, an expert maybe, and do some kind of—if you don't have the resources on your own to do proper maintenance on this. It doesn't mean you just disregard it. You want to bring somebody in who has the wherewithal and can do a proper assessment before the auditors knock at your door.
Jesse: Yeah, definitely because you know as with any aspect of auditing, if you understand the questions and the concerns before the auditor starts asking questions about things, you're going to be in a better position. And so if you say gee I don't think I'm in a good position to answer the questions that the auditors are going to ask, then it's good to reach for help, right? It's better to just be prepared for those things and maybe there—you know there's consultants. There are professional service providers who might be able to help you build up that software inventory, might be able to give guidance on things that are out of date and not supported anymore, right? On IBM i, a lot of people are still running old, unsupported versions of Node.js. And you know it might just be really useful for a second pair of eyes to look at that and say something like, you know Node.js version 8 has been out of support for some time and you should be able to migrate to Node 14 or something that's in support. It should be minimal disruption but they can—sometimes an extra pair of eyes can really help you give—or can really help you get some solid advice and some solid action plans on improving your posture when it comes to security, maintainability, all that stuff.
Charlie: Jesse, we started this conversation talking specifically about Log4j and Log4 Shell, things like that. But for those who might still be dealing with that very issue today, are there any final helpful hints or tips or directives you might want to offer them that might just help them get past where they are today?
Jesse: Yeah, I mean really, it's—again like I said in my blog, it's focused on code that you own, focused on things that are in your control, and stay in contact with any vendors for third party software. You know IBM has a PSIRT page that they continually update depending on where the other vendors are with software that you're running. They usually have similar mechanisms so just continue to stay—keep informed with whatever the software vendors are providing you for information and whatever patches they might have for you.
Charlie: That's a perfect general statement regardless if it's Log4j or whatever is coming up and down the road, right?
Jesse: Exactly.
Charlie: Yeah. Well Jesse, this has been an interesting conversation. I tell you. I suspect that—any final messages you wanted to throw into this conversation? Because I—this is a conversation that could—that is surely worthy of hours of discussion.
Jesse: Hours of discussion for sure. You know I will parrot one of the things I said earlier, which is you know give back to the open-source community if you're consuming its resources. Because that is what allows open-source software to be as innovative as it is, but also what allows open-source software to maintain a good security assessment moving forward is that you have all of these companies willing to give back in some way, shape, or form. And again, that's good advice from a philosophical perspective, tt's good advice from a moral perspective, it's good advice from a good business perspective, even without the consideration of security. But again if you're able to give back, whether that's in the form of developer time or providing good feedback to maintainers or you know in some cases just providing resources for them to run tests against or you know any kind of insights you have that might make this software better, just give back to open source if you're consuming open source. That's just good business and that also will help the overall security of open source in general.
Charlie: Indeed, and just the spirit of the community overall.
Jesse: Yeah, exactly.
Charlie: So, it's all good. Jesse, thank you very, very much for your wonderful insights as always. You always bring such a wealth of information to these topics. I really appreciate your time today and your know-how and your willingness just to continue to share, and how much you do for the open-source community. So thank you very much for that.
Jesse: You bet. Thank you, Charlie—and before we sign off, I want to tell people, anybody interested in open source on IBM i, go to IBM.biz/IBMiOSS altogether, and you'll find all kinds of resources there to help you along your way as well. But Charlie, thank you so much for having me. It's been a pleasure.
Charlie: Always a pleasure, my friend. We'll talk soon. Everybody look forward to other TechTalk podcasts. You'll see them on the website. There's always good topics out there and thank you for joining us today. Bye now.