On December 9, 2021, the Log4Shell / Log4j vulnerability, tracked as CVE-2021-44228, was publicly revealed via the project’s GitHub. Since its release, it has been discovered that the vulnerability impacts many types of software, and likely billions of devices.
The vulnerability allows for unauthenticated remote code execution. Log4j 2 is an open source Java logging library developed by the Apache Foundation. Log4j 2 is widely used in many applications and is present, as a dependency, in many services. These include enterprise applications as well as numerous cloud services.
Join us as the Randori Attack team and GreyNoise research team meet for a deep dive of the vulnerability and how to protect yourself against it.
Featured Speakers:
For immediate insights into your exposure to Log4Shell, read our Attack Team Note. And follow us on Twitter for real-time updates.
If you think you might be exposed to Log4Shell, get a free Log4j Attack Surface Review.
Moose:
Welcome everybody. See the attendee numbers here. Still climbing. We're gonna wait just a couple minutes. I know folks, we've got a lot, got a lot going on. Gonna make a little space for folks to be able to make the jump from their previous meetings to jump in here with us. I appreciate y'all taking the time and we'll get started here in just a moment.
Moose:
All right, everybody. I wanna be respectful of everybody's time. So we're gonna go ahead and get going here. I am. David Woff. Everybody calls me moose. I am the CTO and co-founder here at Rand Dory. I'm pleased to be joined by the team from GreyNoise intelligence. We're gonna be talking about log four J I'm sure that y'all know why we're here. So we'll go ahead and get into it and go ahead and get started on stuff that we're gonna cover today. Usual agenda for these types of things, I'll do a little bit of intro up front. We're gonna try to get as quickly as we can into the technical so that you can get a sense of what's going on. And hopefully we can at least get you all able to walk out of here with a little bit better sense of how to command of the situation, hopefully resolve what's going on.
Moose:
And of course, you're gonna make lots of time for Q and a the end. So if folks have questions, please cue those up in the Q and a. If you have not already submitted them, we'll be monitoring those and try to keep things rolling. Keep, keep the dialogue fun and informational here today, a lot going on whenever we're doing one of these webinars during what I will call a live and exciting incident, you know, the one where everybody's been working all weekend. I like to give you all the bottom line up front, just so that you have a sense of what we're gonna cover. You can get a good sense of whether this is the best use of your time. I assure you we've got a great pay analyst here. We've got a lot of information. I know that Q and a will be excellent at the end, but if you are currently working a CVSs 10 induced fire in your organization, here's the set of things that we're really gonna hit.
Moose:
So if you wanna hop and go patch madly in your environment, and yeah, you are gonna have to make that judgment call for yourself. Everybody TLDR. We're here to talk about this 44 22 8 AKA log for shell. This is in fact a big, big wowy of the CVSs 10. And it's a real one. It's not one of the ones that I downplay. This one's actually a good one for, for attackers to use. It's everywhere. It's being exploited as easy to exploit, and we still don't know everything, but we're gonna give you as much information as we have.
Moose:
Who are we? There are four of us. We're gonna be doing the talking here in addition to anybody from the panel. Well, from, from the audience here, who's got Q and a that we can get into. I am David Woff. Everybody calls me moose. I am the CTO and co-founder here at ran Dory. We do offensive cyber security and attack surface management, as well as automated red teaming. Usually if I'm in a room, I am the hacker or person used to breaking into stuff in the room, given the distinguished panel we got here, I won't take that credit for myself today. I'm gonna go ahead and let our panelists introduce themselves. So you get a sense of who's here, who's on stage and why they are here. And I'm gonna start with you, Andrew.
Andrew Morris:
Hey folks, my name's Andrew Morris. I'm the founder and CEO over at GreyNoise. GreyNoise is a cybersecurity company. That's really focused on watching people who are scanning, attacking and crawling the entire internet. In this particular case, the relevance of GreyNoise is that we operate a gigantic network of passive collector sensors that observe people that are scanning, crawling and exploiting vulnerabilities out on the open internet. If GreyNoise sees something, that means that everybody sees it. And, and with that, Aaron, lemme pass that over to you.
Aaron Portnoy:
Thanks, Andrew. My name's Aaron Portnoy. I run the R and D research and development function here at Randori mainly responsible for enabling our attack team to attack customers. With regard to this vulnerability, I was the person responsible for first identifying that it was bad and enabling our attack team to then go use it and leverage it within hours of it first becoming public and pass off to Remy.
Remy:
Hi, I'm Remy. I'm a security ER researcher at GreyNoise intelligence. And I look at the data cloud from our sensors all around the internet and have been driving the effort and tracking the Log4j related activity.
Moose:
Well thank you everybody. And if it wasn't abundantly clear from the introductions, the reason we put this panel together is because we've had a team of folks who are watching what's going on in relation to the specific vulnerability, as well as a team of people. Who've been figuring out how to exploit using this vulnerability, which gives us pretty comprehensive coverage in terms of understanding what the heck is going on around us today.
Andrew Morris:
So I'm gonna talk about the Log4j vulnerable, the what to log verte specifically. So Log4j is, is basically, it's a logging library. That's used in a lot of, lot of different stuff. So there's always gonna be different complexity that you're gonna be thinking about. Anytime there's a vulnerability that comes up in a piece of software log four J is code used by other code used by many, many applications, right? So log four J is not one application that's used by a bunch of users. Log four, J is a library that's used in a ton of different commercial off the, off the shelf products, open source products, embedded systems, et cetera. And so log four J is really, really commonly used out by and lots of different applications that use Java. One of the reasons that this situation is so particularly hairy, and it's the thing that I actually really wanna highlight a lot here is that log four J is used by a ton of different things, even different products where the author didn't necessarily know that it was being used because it can be used as a, a dependency of another dependency, et cetera.
Andrew Morris:
So log four J is very, very commonly is a logging library that is very commonly used in Java applications. It also uses a number of different, it has a number of different functionality that allows it to basically do things that are kind of similar to treating like Ari fetching, remote resource, et cetera. And that's one of the reasons that was one of the, the cores of the technical bug that we're gonna talk about here today. And it's been around for a long time. And the one thing that you really need to know here is that it's used by a tremendous amount of close source and open source software and a tremendous amount of different devices.
Moose:
So in terms of why this particular bug is such a juicy one to an attacker, why it's very interesting to me, why it's been a thing that we're all here talking about, why we've all just spent and I'm sure everybody in the audience has spent the last, what four days in a panic room dealing with crisis that has come out here is that this particular vulner ability, although it looks complicated in a chart of all of the different callouts and how things move around and where the data goes is incredibly easy to exploit. I'd say on a scale of one to 10, this is about a one in terms of difficulty to exploit for circumstances where it is exploitable, conversely, because of the access that the applications often have, this is a, like a 10 outta 10 in terms of potential impact. So when you're talking about total compromise of some of the applications that we've proved, exploitability of, it's just a tremendous, tremendous opportunity for adversaries to do real, real bad.
Moose:
And then of course, as everybody here has learned, we've been having to make sense over the past several days of all of the various mitigations and alternate fixes compensations that we can apply because there's been pretty limited availability. It patches in the end applications. So it's been a dynamic in circumstance that we're all trying to keep, keep track of. Hopefully we'll be able to make a little bit of sense of sense of this for all of you here. But if you learn nothing else about this, then Ms. Paint graphics can make it to real webinars. Is that this one is a, a real bug. This isn't one of those CVSs tens that's been made out, you know, a mountain out of a hill. This is an actual mountain on its own.
Moose:
Andrew mentioned this earlier, but you may have heard of Java. It is everywhere counting. I think as many devices as McDonald's has served cheeseburgers or something on the same orders of magnitude, right? So when we're talking about the ubiquity of this, it's a tremendous logistical nightmare just to run down every place that this library's ended up, because it can be pulled in anywhere that Java can run. And so it's used everywhere, including man, many massive enterprise products like VMware and jam. And of course it's being exploited actively in the wild. So there's a lot of craziness that we all got go UN crazy as we make sense of the universe over the next coming weeks.
Remy:
So this is a timeline from disclosure to exploit on December 5th, we see that Apache first identifies this vulnerability and very quickly has a turnaround for the first release of a patch on December 9th, GreyNoise begins to observe activity related to log four. J we see a very sharp uptick around the ninth and the 10th as public POCs begin to be available and using clustering on all the different variations of payloads that we're starting to C we are able to certify that payload diversity dramatically increases and WAFF rules start to become significantly less effective. Just recently, on the 12th, we start to see application specific payloads that are targeting very specific applications or containing parameters that only that specific application may accept next slide. This is the same graph as the previous page, but this is grouped by hour. So this large jump that you see is, I'm sorry, this large jump that you see is on December 12th at 9:00 PM. UTC, where we see roughly F five times uptick in attempts from unique IPS within that hour, it continues to grow. And we peaked at 468 unique IPS observed on December 13th at 12:00 PM. UTC.
Aaron Portnoy:
So GreyNoise is great at showing the kind of bird's eye view. I want to put my attackers hat on here and give you a little bit of perspective from Rand's perspective as an attacker. So we do two things, attack, surface management, and then actual adversarial red team engagements. So first thing we did when this came out and I first noticed it at about 3 45 on Thursday, that's when our whole team spun up and we started to get customer alerts and started to map out all the different parts, the attack surface that are susceptible to this, but my sole focus was gaining access to our customers, authorized access, of course, for various targets that we as in an R and D function had been analyzing already. The two that I'm showing here is just kind of case studies and the two that we were successful within hours of the first announcement of this vulnerability coming out were VMware horizon and jam pro VMware horizon was one that we had already been doing research on, but just to give you a data point, it took us five and a half hours.
Aaron Portnoy:
From the first moment we started investigating log four J on VMware horizon to we had a shell on a customer device jam pro, want to call attention to this one in particular, if you're not familiar with jam, this is the software that controls your entire fleet of computers at an organization that one took us about just under nine hours. And that compromise is a game over event for the entire organization. Once you've compromised a jam pro server, you can then push code out to all systems that are under that server's control. So contextually nine under nine hours is how long defenders had before an incident occurred with that particular target takeaway. There is, this is not a race that defenders are likely to win. This is a situation in which you really need to assume that you've been breached at this point. If you have, have any of these pieces of software on your, not, not only the ones listed here, but anything that's using log four, J you really have to operate from an zoom breached situation. Next one.
Remy:
So what are we seeing in terms of active exploitation from GreyNoise point of view, we're primarily seeing payloads being transmitted over HB or HBS using the vulnerable Log4j lookup syntax early on. It started off as all user control, HTP attributes being used to include this vulnerable syntax. So we're talking about the, the path on an HTP request, all HTP, the body, even sub attributes of headers, such as authorization, bearer followed by an injection attack as time continued onward, we see payload, obfuscation and interpolation, which is leveraging the different message. Look up syntax, which is embedded in Log4j for remote code execution. We're seeing loading of remote Java classes, but we're also seeing exfiltration of stuff like environment variables and secrets like AWS secrets through use of DNS. So what started off as homogenous spray and pre has quickly become active targeting of specific app applications.
Remy:
Next slide, these are some examples of application specific pays that we've seen as you can see in the UI. We have vSphere explicitly called out with a variety of headers using confiscation and interpolation to bypass any sort of web application firewall that may be doing its job as best can, as well as a post to in info profile with a content type of application and service ID, which is close to related with another VMware product. Next slide. These are the same payloads as before with the addition of a get request to web SSO, vSphere local. And we start to see people are really digging into this where a specific parameter of this request, the relying party entity ID is a base 64 encoded payload using RMI and is being xFi through port 4 43 outbound. So we start to see some really clever specifically targeted payloads — next slide.
Aaron Portnoy:
So there's been an inordinate amount of discussion out there, and a lot of advice, good advice, bad advice, data that appeared to be true, but is no longer true. A lot, a lot of D just information out there. So we tried to bucket this into effective and ineffective mitigations, and we're only stating they're effective. If we've proven there they've been effective. So I'm gonna run through the effective ones. There are obviously other things that are out there right now, but these are, we are certain about. So with that being said, obviously updating log four J to the latest version will remove susceptibility. If you have the ability to do that, oftentimes with close source applications, you're relying on the vendor to provide updates, but if you're able to updating the version is the clear way to go forward here, then deny firewall rules. This is something that we at rando advise all of our customers to do.
Aaron Portnoy:
When they're concerned about a particular target and exfiltration or exploitation of that target. Anecdotally, I can say that rando has been conducting adversarial operations with this vulnerability. We were not effective on certain customers because of their firewall rules that were in place to stop us. Another potential mitigation here is the JBM. You can actually configure the JBM to not do those lookups. The J N D I lookups. You can do that either on the command line. So again, if you have access to the JBM instantiation and you're able to provide command line arguments, you can disable that look up. You can also do the same via environmental variables. And then we want to address some of the ineffective mitigations. I'll let Remy speak to the first point here.
Remy:
Web application firewalls are absolutely doing their best. What I can speak to in terms of trying to track or tag activity or write any sort of rule set for it is there is a large, large variety and using any sort of static strings is not going to be effective. Updating the Java of runtime itself. As far as we are aware, there's been some mention that being on the Java 1.8 is a safe enough mitigation to the best of our knowledge. The newest version of Java is still vulnerable. If the Log4j library is vulnerable and searching for strange things, such as dollar sign and left curly bracket are likely to be ineffective due to the sheer variety of payloads that we're seeing. Think including far more wild cards and being very, very generic is probably in your best interest and narrowing it down from there.
Aaron Portnoy:
Yeah. And, and just to add to all that, you know, we're talking about web application firewalls here, but I don't think it's really come to light yet. How many vectors of attack there are, right. We've mostly been talking about HTTP here, but I mean, there are quite a few other protocols that are going to be logged in various capacities. I think we're just gonna see that the amount of impact vectors just in continually increase over the coming weeks. Unfortunately. So yeah, web application firewalls only really deal with one small part of that. But I imagine that these vectors are just going to be so diverse that there is really no way to block everything inbound. You almost have to deal with things on the exploitation side or just updating.
Aaron Portnoy:
So as far as remediation, you know, we talked about updating Log4j if you can update Log4j that is the best thing you can possibly do if you have custom 'em applications. So we categorize the remediation into two sections here, custom applications be meaning in your enterprise environment, you've got Java developers and they wrote some application. If that's the scenario, you wanna identify everything you can, that's using that library. You wanna upgrade them if you can, if you cannot, you can use some of the mitigations that were mentioned prior in terms of enabling J and D I lookups, as I mentioned earlier, you want to be working from an assumed breach situation, scenario perspective at this point. So monitoring systems that you have identified that are running Log4j you are looking for bad behavior that is already on those systems, not necessarily future bad behavior that may be on that system.
Aaron Portnoy:
Implementing default denied policies is general guidance that we give out on most targets. And this would certainly prevent exploitation of this particular vulnerability being that it requires a request to go inbound, and then the actual exploitation is reaching back out. So if you're able to implement firewall policies to block that, we've seen that to be very effective. And then of course, checking logs for signs of exploitation. Remy kind of mentioned that that's not maybe the best way to do it. It is certainly something as part of a defense in depth that you should be also watching all your logs. So that's talking about custom applications. The other section here is third party software.
Aaron Portnoy:
Third party software are being, you know, VMware horizon or jam or other software like that, maybe close source. So if you have access to the systems that that software is running on, you probably want to do something like scan for the presence of the jar files. So on our blog post, we have hashes for all those jar files that are susceptible. You can search for those. If you identify those on disk, you can assume that they, you know, that system may be susceptible. You can also ask the vendor for information, although they all tend to be very, very busy right now. It's very, it's in our experience, it's been very difficult, getting answers from various vendors about the status of their patches and whether they themselves are compromised, which is difficult to deal with. And then again, the last two bullet points here are just general guidance that we would say, whether it's a custom application or third party software default deny, keep an eye on the logs, get as much telemetry from those systems as you possibly can. And look for Arant behavior.
Moose:
Hey, Randy, Aaron, I'm gonna come over the top for a second and just add on to the back here. After you've done whatever defenses you're gonna do, whatever remediations you're gonna do. If you're fixing or updating your firewall, test that from outside of your network and confirm that those protections are actually working as you think they work. There's a tremendous amount of variability in actions. Some of the configuration parameters, for instance, only affect certain versions of Log4j. So after you've done the thing that you think fixes it, it better make sure that you've actually gotten that fix in place.
Moose:
So I'm gonna pause here. We blast through a bunch of, of content. I know that there's a tremendous number of questions that folks are beginning to tee up in the chat. So I'm gonna invite folks again, you know, if you have questions, please feel free to drop those in the chat for us. We've got a team who's following. I'm gonna throw a couple of the additional notes to you. I know GreyNoise has got a tremendous blog. Andrew, I don't know if you wanna share a couple thoughts about the types of things that you've been following in the observations here.
Andrew Morris:
Yeah, so really quickly. I mean, one of the things that was interesting about this is how quickly it went from the world, kind of becoming aware of this vulnerability to the entire internet, getting hit with exploitation of this vulnerability. This one was really quick. And so that was one of the things that we found to be really interesting. As Remy mentioned before, this one is, as I know everyone is painfully aware right now, nightmarish to detect on the wire because of all of the interpolation that you can do in all the ways that you can make the payload look completely different and still be functional. So that's one of the reasons that this is kind of disaster Remy mentioned this in the, in the, the timeline, but seeing the exploit attempts go like really basic stuff inside, like in strings that are flying by to things that were, you know, basically quickly diversifying to the, to application specific things and things that were clearly supposed to be bypassing maps was interesting.
Andrew Morris:
One of the things that we saw that that's really interesting about this one right now is that currently AB over half of the IP addresses that are actually actually spraying the internet with this exploit payload in an attempt to figure out the blast radius and how many people are affected over half of the IP addresses that are probing for this are actually benign IP addresses. They're actually security companies, which is something to just be really aware of. We're seeing a lot of security companies that are obviously like, you know, blasting that around the internet to try to figure out how bad it is. It's really tricky. I don't wanna add, you know, guidance that I think is untrue. And I don't wanna tell people something that is not something that I believe to be certain, but it's possible that of the basically IDs alerts that are going off of this flying in, there's a really good chance that some of those are security companies, as well as, you know, obvious, bad guys that are trying to get in on this.
Andrew Morris:
Some of the other things that I've found to be interesting were just that we've noticed that the amount of IP addresses that are attempting to broadly exploit this on the internet has been over doubling every day. And so we're still seeing that kind of continue to go up philosophically. We kind of have found that the more of vulnerability pays off to a, to a opportunistic attacker, the more traffic it's gonna get, we're gonna see from that. And so as there are still more and more people that are vulnerable to this, it's probably gonna continue to go up and as more and more people start to get patched, what GreyNoise is seeing is gonna slowly go down. And so those are some of the quick things that we've found that are, that have been interesting about this. Somebody specifically asked in about, you know, the different handlers that are being used.
Andrew Morris:
Someone asked about L D a about X, basically xFi happening over L D over STL. We are seeing that almost every which combination that this can be exploited. We are seeing happen at internet scale right now, which is one of the things that's, what that really means is that we're watching in real time, as a lot of the bad guys are starting to figure out new, clever ways to exploit this vulnerability. And it's clearly paying off because that's one of the reasons that people are still doing it. Remy. Do you wanna add anything that you found that was, that was interesting from your observations over the weekend or at right now?
Remy:
Absolutely. There was a very distinct time late on Friday night where the variations and payloads pretty much exploded and any sort of rule set or, you know, static string checking kind of went out the window. We're seeing because of the simplicity of attempting this vulnerability. We're seeing people take off the shelf tools for internet wide scanning and just stuffing every single header possible with a, you know, payload attempt. So in terms of grouping this together, it quickly became the only viable way to kind of break this down so that a human could take a slice of this data and look at it to start to use clustering. Otherwise you just have such a extremely large variation. You don't really have a hope of understanding. So as we start to break this down, we start to realize the scope and variability of it. I do wanna reiterate that blocking outbound for DNS or RMI or any known protocol. If you, if you don't know what your server's reaching out to and have it explicitly allowed it dropping it as a safety measure is probably a good idea.
Moose:
That's an awesome segue. So I'm gonna push us into a little bit of Q and a, if you haven't done. So follow both of these cool teams on Twitters, as well as I follow the ongoing blogs, we both have going, we got links in the chat for all of that. I'm gonna take Remy's cue and I'm gonna pull my current favorite question from the Q and a to the top and take also the first crack at answering it. My favorite question is that, Hey, what would we block? What do you mean by default? And I, what is this thing that you're describing? And what, what we're actually talking about here is, Hey, you're, you have critical perm systems that quite often are only initiating outbound connections to a small number of resources. So I'm thinking of like jam or your VMware applications that accept incoming connections.
Moose:
So established related incoming is good, but they don't talk out to a lot of things. And so if you just tell you your perimeter firewall outside of that, Hey, don't let this box talk to everything on the internet. Don't let talk to anything on the internet, and then only let it talk to things that it needs to. So maybe, you know, if, and remember that shades of gray here are good, right? It's not, it's not binary, but as a great example, we've got folks who only allow their, you know, perimeter observers to talk to the net net block ranges associated with updates and that everything else is rejected for outbound. And then they alert on any anomalous connection. In this case, if you blocked an outbound connection from your jam or from your VMware V center, you stopped the exploitation at attempt, even though the trigger had occurred in the, the log parsing itself. So that's kind of one of the things that's definitely worth doing. I always tell people to do default deny for every perimeter system, you know, anything that's in what we use to call DMZs, like restrict what they're allowed to talk to. And yes, I'm fully aware that that gets harder as we have more integrated services online, but a little bit of default deny goes a long way before I jump to question two, since I answered that one, myself shamelessly, did I miss anything on the, what would you actually block Remy Aaron or Andrew?
Remy:
Nope, not all. I think you covered it actually. Yeah.
Aaron Portnoy:
Awesome. Yeah. I was gonna say, I want to try to do anything specific because this is such a fluid situation and all the methods are going to be changing. So you need something general and fundamental.
Moose:
Excellent. So there's a bunch of questions that we're getting around versions that are vulnerable. Would anybody here wanna take a first crack at what's going on with the different versions of Log4j which ones are and which ones are not vulnerable? And what do we know?
Aaron Portnoy:
Yeah. We've been looking at that internally here from our R and D team, the there's conflicting data. So initially there were questions about log four J version one. The first mention of whether it was vulnerable was from a developer on the actual poll request on the GitHub for this vulnerability. And he said, yeah, it looks like it's probably vulnerable. That has evolved and changed to the point where I think where, where we've settled at this point is, is it's. I would assume it's vulnerable. It is likely only vulnerable in particular configurations. And no one has yet run to ground specifically what those are we're hoping to do. So here shortly, keep an eye on our blog. We're gonna be updating that as soon as we have a conclusion, but we intend to provide some clarity on that. But generally speaking, I would just go ahead and assume anything logged or J should be updated at this point,
Moose:
Andrew Remy, any additional flavor from, from your perspective there. Alrighty. Hey, so another good question here. I always love the, what if and what's coming series of questions, but couple of folks threw at us, Hey, is this gonna be one of those things that gets to be wearable? Is this gonna be a big one? Remy, why don't you take the first, first one there,
Remy:
This was a practice question that one of my colleagues raised in like a mock run through of this a few minutes ago. And love it. One of my colleagues, Paul, a article by research team explicitly and clearly calling out. Yes, there's already examples of it
Moose:
For those who might not know, does anybody wanna define what that means? Like what does it mean to be wearable? And then what would the ramifications be
Remy:
To be warm from my understanding is when an successfully invades the system, it kind of sets up shop and continues to scan both internal and external networking to redeploy the exploit to another node
Andrew Morris:
And on and on.
Aaron Portnoy:
Self replicating or self propagating. Yeah. And I think that that's gonna happen more and more and more considering, like I said earlier, the amount of vectors, like we're only really dealing with HTP right now, but once people start to figure out, oh, Hey, job is on embedded systems and maybe industrial control systems. And maybe on, you know, the, the, the projector system in your corporate office, like the you're gonna start to see more and more weird worm, more mobile type things going on for sure.
Moose:
Awesome. Another good question here, folks wondering about like, Hey, if I don't have a, you know, external volume scanner, is there some research or so tool that I should be using? How do I actually do the thing that moose you said to do and check the, the patch or fix that I put in place is working
Aaron Portnoy:
Depending on your level of technical aptitude or how comfortable you are with some of these tools. I've seen a, a burp, if you're familiar with a burp suite, there's a, a nice plug in there that is supposed to be kind of a, an easy to use passive scanner that will kind of fuzz your services. As you browse. There are a number of software or tools that have been released by various companies that do a, a, some subset of fuzzing. The issue with them is that they're mostly testing for very rudimentary exploitation attempts. So they're, you know, as, as Remy was saying, they're indicative of a lot of the early attempts that GreyNoise was noticing, right? Just stuffing data and payloads into host, you know, HDP hazards. So there are some software I can drop some links to those. Some, some tools that people have put out, I'll drop that in the chat here in a minute,
Moose:
I'll do the shameless plug as well. It rando does attack surface management as a solution. So one of the things that we do is try to find products on customers. Perimeters, if you are struggling with knowing what's on your outside, we reach out, we can at least do a free scan and, and go from there. We've also been developing probes for the vulnerabilities we're talking about here. So, so my, my one moment of commerce,
Andrew Morris:
I'm noticing a question here in the chat, on the risk to vulnerable internal systems. I actually wanted to read this one off specifically, because this is something that we had to think about at GreyNoise, because we're moving a lot of data around and there's a lot of serializing and deserializing that's occurring in a lot of different places in, in our data pipeline at a glance. It's really kind of tempting, I think, to say like, well, if something's buried inside your V your, you know, your internal network and it's not talking to the internet really, and there's nothing talking to it really, is it vulnerable? Unfortunately, the answer is it's worse than that, because data can travel around a and if data is moving through a, a data pipeline, a logging pipeline files that are moving around from one system to another, then yes, there is still risk to internal systems.
Andrew Morris:
It's just not going to happen the way that it's happening. This that's this straightforward where basically, you know, you've got a service that's running on a perimeter or something like that. It gets, you know, an exploit attempt and then it fires something back out. We're already seeing some like really crafty ways that people are trying to get the vulnerable payload to basically move throughout complex distributed systems. So I want to, I wanna make sure that I'm sticking to things that are, that are definitely true and that I'm not giving, creating any additional Fu so I don't wanna freak people out anymore. I wish that I could say that if something was on an internal network, it's not vulnerable. Unfortunately that's not true. If something is on an internal network and it's vulnerable, it is still a risk. So that's just, that's just how it is. Unfortunately.
Aaron Portnoy:
Yeah. On that note, we've actually seen callbacks that occur much later than when they were thrown. So someone throws an exploit and it could be six hours later. You get a call back because of what Andrew's mentioning, like maybe it's a log rotation. Maybe it's a, an R sync that happens like the data's being moved. It's, it's a, it's very asynchronous.
Moose:
Couple of questions popping up from, from fan and favorite commentators about the attack and patch script, and O cyber reason released a tool that you can use to attack yourself and kind of close the door behind you. Has anybody on the panel had a chance to look at those and thoughts on effectiveness, how to think about those types of things.
Aaron Portnoy:
Yeah. I looked at that cyber reason vaccine, they called it, which was, it's a cool idea. This has been done before. It's basically use the exploit, use the vulnerability itself to patch the vulnerability. What they do is they, from my quick read of the code is they use the vulnerability to dynamically in memory, essentially remove references to the class that is susceptible to future exploitation attempts that is likely effective. However, it is a runtime hot patch mitigation, meaning if the, the job of virtual machine or the system or the application reboot or shut down or restart, you would have to rev vaccinate. So it's only really good for the lifetime of the application. I haven't tested it myself, but from looking at the code, that's what it appears that appears to be a limitation of it.
Moose:
Yeah. That's awesome. Also just remind folks that, you know, saw somebody else remark that if somebody closes the door behind them, you might not know if you're, if you've been exploited, it is definitely worth going through the legwork. If you have a system that you know, is logged for J it's on the internet, or it's in a path of something from the internet that, you know, assume, assume bad, and then work your way back from there. You know, I, in past engagements have closed vulnerable holes behind myself. So I wouldn't put that past the adversaries here. Other piece when you're trying to fix these things dynamically like that, there's a lot of variability in the configurations and environments. And so the efficacy of those techniques might vary dependent on the particulars of, of the environment you're running in. Go, good question, Remy that I'm gonna throw at you. Is there anything unique about earlier exploitation or exploitation attempts that you saw versus what you're seeing today? So, folks, I think the spirit of the question is if people are worried that they may have been exploited early, are there particular IOCs that they could go looking for, or other indicators, ways to think about stuff maybe before the ninth versus after the ninth,
Remy:
In terms of something I could say out loud, no one wants to try to say red rejects out loud, but in terms of the variability of the payloads, they, they really, really did explode. But for early exploit attempts, everyone kind of seemed to be riffing off of a basic J NDI colon L D colon slash slash as a static stream that you could grip for in your logs or query stuff. So that was a fairly good baseline and kind of getting a grip on what's in your logs, kinda like what's happening. And there was a large enough volume that if you have a server on the internet, you will probably find something querying for that string. But as I mentioned earlier, with the large, large variance that really late on Friday night, you're gonna want to use a lot more wild cards in terms of querying for what you're looking for. Like the letters J NDI stick, a wild card on both end and between every single letters and start narrowing it down from there.
Moose:
Awesome. Hey, I'm gonna throw this one at you port, right? Does this vulnerability get you outside of the Java VM? Like what kind of access does this vulnerability give you if you're successfully, if you successfully exploited?
Aaron Portnoy:
Yeah. So the mechanism by which the most exploits are leveraging this is that you're forcing this job application to reach out and get a class file. So that's a bite that's compiled Java code, right? Bite code, what that class file contains and what that payload contains is entirely up to the attacker. What we've seen people doing. And I assume this is the most common thing that people are doing is basically just providing a system exec. Think of it like basically, what, what commands do you wanna run on the system? And they're gonna use Javas runtime dot exec to execute those on the system. So basically you're going from a class file that gets shipped down onto the vulnerable system. That Java class file gets executed. Currently. That seems to be people running a command like, okay, run, curl, or w get or something to pull a second stage payload down that is no longer Java, but probably a binary and implant of some kind. However, it is entirely Fe that someone could keep their payload entirely in Java. So that's really a dealer's choice as far as how the attackers choose to go about exploiting it and leveraging it right now. I think most people are just going straight from the Java class file to a system command to a second stage payload, maybe GreyNoise guys, have some more insight into what you've seen
Andrew Morris:
On now. Let's see the PA we haven't actually seen anybody try to use, hang on, Remy, can you confirm this? Have we seen anybody at least from the opportunistic exploitation side, do anything out? Have we seen anybody try to remote load remote Java classes from in our opportunistic exploitation side? Cause I've mostly seen like system calls, calls to w get calls to occur and other things like that, but I've not, I don't believe we've seen the remote Java classes. Have you seen that?
Remy:
So those commands are effectively loaded in the class. That's returned from the server when to, to speak to your point. No, I have not seen any specific exploitation of those Java classes. And part of the reason for that is as this really started to explode and, you know, waking up and the, the, the Internet's on fire, we, we made a conscious decision to focus on what, what, we're good at the data tracking the activity go, as far as fetching payloads, investigating, starting to look at what attribution might look like quickly became infeasible. And so we focused on getting the information that the community needed at this point in time out into the open.
Aaron Portnoy:
Yeah. So what I suspect you'll probably see is all the incident responders who have, have been called in over the weekend and this week, who are all doing all that hard work of identifying all the payloads and what actually is going on. You're probably gonna see a lot of writeups in this coming week about, Hey, they did this payload and here's a different version. But as far as what I've seen so far, people are going quickest path path to win. Quickest path to win is execute a w get and put a thing down. And you're done, yeah.
Moose:
For have led some additional commentary that there are prior to this vulnerability being disclosed. There were Java serialization frameworks that pen testers and others had used for conducting attacks. Most of what I've seen are indicative of, of folks who are doing scanning or attacking that are based on that, that work that's been in the public space. It's a lot of execute. She command as kind of the end game to get to whatever the next stage is. Would anybody wanna comment about the particular protocols actually involved here? I know that there's a lot of folks, I think, who are trying to scratch at what they might try to filter or block or investigate based upon, you know, saying that there's, you know, string in your logs that causes Java classes to be loaded. Sounds kind of scary. What's the, what's the TLDR on what's actually happening and you know, what kind of protocols would somebody look for? Or what, what, what are the steps involved here
Aaron Portnoy:
So that information's still coming to light? I think what we've, what we've seen, what we've used are, you know, you have the J N D I colon, and then you have the, the protocol that you choose to use L DAP was the first one. I think that was disclosed, which is the lightweight directory access protocol. So you that's the protocol you would see going outbound from the vulnerable system. That's being exploited out to the attackers system. You were also seeing RMI, which is slipped in my mind now, remote remote management method, remote method indication. Right. I can't recall, but it's the, it's the job of promoting protocol, which there's an RFC for I'm sure. And you can look that up. The other thing is there, there are quite a few other supported. I schemes I'll call them. I've seen, I think was NDS. I, web logic has its own. I can't recall what it's called, but there, there are at least six or so different protocol schemes that I've seen. I'm sure GreyNoise is gonna have a lot better telemetry on that going forward through the week. But there are a few that I've seen L DAP and RMI are the two most common, but there are at least four others that I've noticed.
Moose:
Yeah. And I know I'm seeing some questions in some clarifying points in the chat as well. There are a whole suite of remote method, invocation tools in the Java tool logs, but only a small number of them are supported by the lookup interface used by Log4j. And so in the kind of Log4j world, the two big things that we see are L D a and RMI. I think the LDAP is much more applicable for the older, older Java versions. And we see the, the RMI used more prolifically on the, the newer stuff, but from like a, like what you would see and what you would block, obviously, if you've got your WF at the front, you've got HTTP or other protocol, it's gonna be really hard. We've talked about all the reasons why that can be very difficult here. And I'd love to get Remy back on a soapbox, explaining about why this one's gonna be difficult there.
Moose:
Once that, you know, malicious string is made into your logs, you're gonna see at least two things that happen. One will be probably two things will happen. One will probably be a DNS look up to go figure out like, Hey, where is the server that I need to go get my malicious payload from? And then the actual call out via RMI note that the attacker can control both the server and the port used for that, that connection. And then that will basically Des serialize an object on the far side, on the client, on the victim side. And that's where we'll see the execution of a typically a shell command to do second stage access there. And then the protocols involved beyond that are gonna be whatever you want. So you have kind of two opportunities for outbound connections that you could block. So DNS, as well as TCP connections, outbound.
Aaron Portnoy:
Yeah. Speaking of that, DNS, one, someone in the chat just mentioned this and thank you for providing the data. I couldn't recall some of the other protocols. I, I, I O P CORBA, but to the DNS point, you know, there's the most severe impact here is remote code execution. You think, right? And that's what most people are doing. But moose had mentioned DNS is possible here. So what you're seeing is a lot of people exfiltrating environmental variables from the Java application via DNS. So if they use the DNS as the outbound, you can set the sub domain of the DNS query to be the environmental variable that you wanna steal. The impact of that is the impact of what that environmental variable can do, which can be just as significant as RCE. In fact, it could be RCE on many things, or it could be complete access of PII. It could be all sorts of really devastating impact.
Moose:
Yep. AWS keys, the database passwords to system administrator passwords have seen all of these things in clear text and environment variables going out. So lots of, lots of opportunity there.
Andrew Morris:
I wanted to have one, one more quick note. We've, we've been trying to do our best to keep up to publish the payloads. The raw payloads like GreyNoise is actually observing at scale. This is tricky for a number of different reasons, but we're gonna try to keep getting these out to people just so that, you know, honestly, the doing pattern matching on the way in for this bug, we've all kind of figured out is not going to be the best way forward for, for catching it. But a lot of people have, you know, networks that allow them to go back in time and just look to see like over some period of time, whether or not something came in was successful. And they may need to know exactly where something was talking out to. So we're gonna try to do our best to at least keep rolling through with the payloads that we find and, and reporting those out to folks in the event that they are useful. So I just wanted to make that, make that, make folks aware of that. You can check it out on the GreyNoise blog for the details on that.
Moose:
Awesome. Awesome. Yeah. I have another question that popped in that was around risk to vulnerable internal systems. I know we touched on this a little bit, but this one caught my eye, cuz it says I've seen proof of concepts using emails, JavaScript to internal users in order to spray internal systems, which I guess technically isn't, didn't have a question mark at the end, but I will just remark that literally any way that you can get a message into a log string, be that an HTTP request, which you can do through JavaScript in the right circumstances. There's lots of ways to get probes to internal networks from outside as well. So, but literally anything you can do to get a message into a log has potential for impact here. So there is, as Andrew said earlier, a non-trivial risk to internal components, but if you wanna show or link to any examples that you're specifically looking, I'd be happy to look at those as well. Good questions.
Andrew Morris:
There. Yeah. That one is that one jar. When is the storm gonna hit and how bad is it gonna be? It's always risky when people speculate this kind of stuff, right? Because being, being wrong is much more likely statistically things just don't ever go the way that you want them to go sticking to the things for sure that we know are true. We've seen bugs that are kind of like this before that have problems that are kind of, I'm reminded a lot of har not of har bleed of shell shock actually some time ago where there's just, you know, a ton of things that are happening in bash in bash environment variables that are, you know, executed. And that was when we all learned just exactly how many things go into an environment variables from, you know, web servers, et cetera, stuff like that. I'm a little bit reminded of that.
Andrew Morris:
I'm reminded of a couple other ones. The good news is that everyone's like, everyone cares about this right now and is like really swarming on it. There's gonna be like the long tail of things that are just gonna take a really long time to get to, but it's it in my experience, your rare early, correct. When predicting the shape of the bad thing that's gonna happen. So I tend not to the, the most likely thing that is going to happen is that I think basically based on where we are right now, sorry, it's not the most likely thing that's going to happen. That the way to think about it is much more of, in my opinion, like how can I get safe as soon as possible? And how can I mitigate the most bad things as soon as possible? Don't think about the big, bad thing happening. Cause there's nothing you can do about it.
Moose:
Go ahead, miss. I, I always like to remind folks that there's a lot of stuff to be very positive about. You know, as Aaron mentioned earlier, we had a pretty high percentage of our ran attack customers who were successfully defending against attacks, leveraging this capability. And that was based on, you know, pretty aggressive DMZ strategy, default deny. And you know, being able to detect can respond quickly. When I think about something like this, you're really treating, you know, a vulnerability that's in the public space is if it were a zero day, right, you don't have a patch available yet. So what are all the other things that you can do? And what are the, all the other things you already did that, you know, help build the resilience institutionally against these types of capabilities. I recognize that most folks are gonna be in firefighting mode right now.
Moose:
And it's gonna be great for people to implement remediation specific to this vulnerability because is you have the institutional will to do that right now. But it's, I think even more important that you can get that extra half step to say, okay, what class of vulnerabilities is this thing reflective of? And which defenses have I implemented that are gonna help me against that whole category of, of threats. So that the next time this happens, we're in a little bit better shape and we're not just, you know, chasing the patch again. And there are some, you know, fairly fundamental things that we can do that are, are pretty, pretty effective. So highly encourage folks to figure out how much defense in depth and how much default deny they can possibly do. Cuz a little bit goes a really long way is something to be positive about. Cause there are folks who are, are yeah, egres filtering and, and allow us for the win. For sure. We are coming up on five minutes to the hour. I'm gonna do a round the table here. Ask if we've got any closing remarks from our panelists who I would like to thank for spending a flurry of hours prepping for a conversation. I know everybody's working short notice. And so I'm gonna go to Remy you first. I'll code to Aaron. You second. And then Andrew, I'll let you get the, the last word in here. And I'm gonna ask each of you to try to think of something positive.
Remy:
Something positive. If you see an exploit attempt, it'll be in the logs. Yeah. This is a CVS S 10 and complexity is very, very low, but one of the most positive things that's come out of this is the, the sense of community and overwhelming support and sharing inform coming out of this. It's it's been absolutely wonderful so far and we're gonna get through it together.
Aaron Portnoy:
Something positive, huh?
Moose:
Yeah. Aaron, Aaron, what have you got? Well, your closing thoughts and that I'm asking you to try to do something positive. Yeah.
Aaron Portnoy:
Alright. Closing thoughts are, there are two main things that everyone should be paying attention to just generally with this as they, cause this is a super fluid dynamic situation. To me, the two things that are most interesting about this are what are all the things that are affected because of the ubiquity of the D diversity of targets and things. There are a lot of different efforts going on from various individuals and companies and government entities to try to get a handle on what are all the things, what is the, can we come up with a master list of everything that's vulnerable, that's something to pay attention to. And then the other side of it is what are all the vectors? Cause I keep harping on that, that everyone's focusing on, you know, some of the easier protocols, but there absolutely are going to be some very strange looking vectors that will be emerging, something positive.
Aaron Portnoy:
I just wanna echo what Remy said. I've never seen so many people hop on this type of a issue before and really share information and collaborate and do so in a positive there's no, not a lot of grandstanding. Everyone is legitimately understanding and empathizing with how bad this is and how bad it is for defenders and being that I'm an attacker. I have a bit of a unique perspective on it. That is makes me excited, but I have to keep in mind, you know, there are a lot of people who were not as excited as I was this weekend,
Moose:
A hundred percent Andrew take us home.
Andrew Morris:
So this vulnerability is forcing us to have a lot of really important conversations with ourselves that we need to have. I think one of the things that I feel really encouraged by is that people care a lot and that it's kind of made its way to mainstream. And so there's kind of a lot of people, even outside the security community that care about this, which is great. I think our job should be in part to make sure that people outside of security have to care about security as little as possible. But I like that people do care in that way. I think we're forcing to think about the software that we run and kind of the, sort of the w the software that our software runs on top of and the budgets and the resources of the people who maintain that software. And so I think if there's one good thing that could come out of it, it's that we all would say, maybe we should invest more in a community.
Andrew Morris:
And, you know, in the open source developers who are writing and maintaining some of this stuff, to make sure that they've got the resources and the time that they need for, to, to push things out. So that's something that I'm really encouraged by. And I say that there has been very little there. I think there is, there have been very li very few vendors that are attempting to really slimly opportunistically take advantage of this, that don't have anything useful to add. That's that's happened. I think very, in a very limited way. So I've been really encouraged by the response and security community. I would just end by saying that we still don't know a lot of the things that we don't know, the situation is very much developing right now. I can actually feel the fog of war on this as we're still going through. And we're, we're doing our very best to only say things that are true, but you know, in a, in a big way, this is this one is big and har and complicated. So thanks for sticking.
Moose:
Awesome. Andrew, me the entire Grano team. Thank you so much for everything that you're doing for the community here. Aaron, thank you for joining us. Appreciate you representing that render team in the Hawk. If anybody still, still listening to us, check out the Twitters, check out the blog posts and then more to come. We'll keep you updated with as much calm, factual stuff as Andrew and I can wrangle our teams into putting together. Thank you everybody for joining us and look forward to a day that we're not meeting for this reason.
Andrew Morris:
Thanks again. All right. Thanks so much folks.
