Back to All Blogs

Maxim Gofnung


Responding to the Log4j Vulnerability (CVE-2021-44228)

December 16, 2021

What a week it’s been. We hope that you are well and expect that you’re ready for some rest with all the Log4J work that’s been required over the last week. We wanted to get you an update on progress and a deep dive on our response. There’s nothing required on your part if you are a customer – your collections have been automatically recurring and ANY positively identified endpoints will surface an issue in the issues page. If you have no Log4shell issues, we have not identified a vulnerable endpoint. If your collection does not automatically recur, you will want to refresh to check for vulnerable instances. 

As you may know from our direct communication, we’ve been scanning for just over a week now, and have identified many vulnerable endpoints in customer environments. We’ve iterated our checking logic continuously and now are on revision four of the check.

We answer some common questions below

How is the vulnerability exposed within applications?

Compared to traditional single-product vulnerabilities, where the affected entry point is known, CVE-2021-44228 aka Log4Shell stems from a library that can be difficult to identify, or less exposed than the application's main functionality. This is, of course, reminiscent of the widely exploited Apache Struts vulnerability.Each respective application using this library can be affected in its own unique way, entirely based on how the library is used. 

To share an example, we can take two different web applications that use log4j as a dependency. Application A is a sensitive banking application in which verbose logging is performed practically on every portion of data sent to the application. While Application B on the other hand is less sensitive by nature and only logs cases whenever an error has occurred within the application logic. Triggering the error in Application A would be as simple as sending any arbitrary HTTP request that contains the payload. However, repeating the same action on Application B and saying it’s not vulnerable would yield a false negative as the logging is less verbose and only occurs within specific circumstances. 

How does the check work? 

With this in mind, a solution was needed that would be able to perform the vulnerability check at scale (in terms of millions of hosts) in a timely manner while reducing the chance of missing false negatives. To meet these requirements, the ASM Log4Shell vulnerability check was developed to work as such: 

A total of 4 different HTTP requests are sent to EVERY endpoint, these include:

  •  A "Fat GET" HTTP request in which the payload is in the URL path, HTTP headers, and the HTTP Body.
  • POST HTTP request in which the payload is in the HTTP Headers and the HTTP Body.
  • HTTP request using an invalid method that contains the payload in the URL Path, HTTP Headers, and the HTTP Body.
  • GET HTTP request which contains the payload in the URL Path and the HTTP Headers.

The intention behind sending different types of HTTP requests is to potentially trigger various behaviors in the application which could lead to the logging functionality being invoked. 

Each payload is sent in the following HTTP Request Headers:

  • User-Agent
  • Referer
  • Cookie
  • Via
  • Origin
  • X-Forwarded-For

Every time a payload is included in a request, it is uniquely obfuscated to help reduce the chance of the request being rejected by a WAF. This means that a single request can have up to 10 different variations of the payload.

In cases where the payload is successfully evaluated by a vulnerable application, it will exfiltrate the running Java version via a DNS Lookup to a service under our control. During the first rounds of writing the vulnerability check, we discovered that in addition to finding vulnerable instances,  third-party services such as Cloudflare would follow the DNS Callback URL regardless of whether the application was vulnerable. This resulted in a high volume of false positives. By extracting the affected application's Java version, we can confirm it is in fact vulnerable, and this is our current scanning behavior. 

Once the check is run and identifies a vulnerable application, the following will be included in the proof:

  • Java Version extracted from the vulnerable application
  • IP Address of the DNS Resolver which invoked the DNS lookup
  • From where the payload was evaluated within the HTTP request
  • DNS Callback URL 

The information contained in the proof is designed to help make the remediation efforts easier, and we hope that this information can clarify where the automated checks are occurring. 

This check runs against every single endpoint as we automatically discover it today. 

What else can I do to find additional instances and entry points?

Based on the discussion above, you have probably already realized that our approach will lead to false negatives, and that additional manual checking for vulnerable products is required. In order to make this process easier, we provide the following search, which can surface assets with known vulnerable, or suspected vulnerable technologies. Note that this search does NOT account for version information, that must be inspected and vulnerability must be confirmed on a case-by-case basis. 

product:Druid product:Flink product:Solr product:Kafka product:"CAS Server" product:Jira product:Confluence product:Bamboo product:Crowd product:Fisheye product:Crucible product:"Symantec Endpoint Protection" product:WebEx product:"Cisco Integrated Management Controller" product:Umbrella product:"Cloudera Data Science Workbench" product:"BIG-IP Access Policy Manager" product:Splunk product:vSphere product:"vRealize Operations Manager" product:"Workspace ONE" product:"Unified Access Gateway" product:"NSX Edge Load Balancer" product:"Horizon View" product:lumberjack product:Kibana product:MobileIron product:Tomcat product:Struts product:Java product:"Weblogic Server" product:"Java Servlet Container" product:"Glassfish Server" product:"Fusion Middleware" product:"Mojarra" product:"iPlanet Web Server" product:"Dynamic Monitoring Service" product:"Glassfish" product:"PeopleSoft"

This search will be actively updated for the next month as additional information is shared and we add additional product signatures,