1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09

2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt

3. The package is in quarantine on pypi - this blocks all downloads.

We are investigating the issue, and seeing how we can harden things. I'm sorry for this.

- Krrish

- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted

We are still scanning our project to see if there's any more gaps.

If you're a security expert and want to help, email me - krrish@berri.ai

What about the compromised accounts(as in your main account)? Are they completely unrecoverable?

And

> Dropped you a mail from [email]

I don't think there is any indication of a compromise, they are just offering help.

The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI. We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow. All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri. Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.

From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages.

The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately.

We are actively investigating full scope and blast radius. Our immediate next steps include:

reviewing all BerriAI repositories for impact, scanning CircleCI builds to understand blast radius and mitigate it, hardening release and publishing controls, including maintainership and credential governance, and strengthening our incident communication process for enterprise customers.

We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.

It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does.

It seems like software tests need to have a class of test that checks whether any of the components of an application have been compromised in some way. This in itself may be somewhat complex...

We are in a world where we can't assume secure operation of components anymore. This is kinda sad, but here we are....

It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."

Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.

Personally, I am a heavy user of Firejail and bwrap. We need defense in depth. If someone in the supply chain gets compromised, damage should be limited. It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.

But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges.

> It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.

Neither of these is easy tbh. Entering a Linux namespace requires root, so if you want your users to be safe then you have to first ask them to run your service as root. eBPF is a very hard boundary to maintain, requiring you to know every system call that your program can make - updates to libc, upgrades to any library, can break this.

Sandboxing tooling is really bad.

(And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)

That's true, sort of. I mean, that isn't the whole point of sandboxing because the threat model for sandboxing is pretty broad. You could have a process sandbox just one library, or sandbox itself in case of a vulnerability, or it could have a separate policy / manifest the way browser extensions do (that prompts users if it broadens), etc. There's still benefit to isolating whole processes though in case the process is malicious.

> (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)

The only linux namespace that doesn't require root is user namespace, which basically requires root in practice. https://www.man7.org/linux/man-pages/man2/clone.2.html

Podman uses unprivileged user namespaces, which are disabled on the most popular distros because it's a big security hole.

Not really.

Let's say I am running `~/src/project1 $ litellm`

Why does this need access to anything outside of `~/src/project1`?

Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.

> Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.

Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.

Not really. I try to capture the most common ones for caching [1], but if I miss it, then it is just inefficient, as it is equivalent to a cache miss.

I'll emphasize again, "no linter/scanner/formatter (e.g., trivy) should need full disk access".

1 - https://github.com/ashishb/amazing-sandbox/blob/fddf04a90408...

Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy. Are cache volumes shared? In that case, every container is compromised if one is. All of these problems and many more go away if the project is built around them from the start.

It's perfectly nice to wrap things up in docker but there's simply no argument here - developers can write sandboxes for their software more effectively because they can architect around the sandbox, you have to wrap the entire thing generically to support its maximum possible privileges.

Inside the sandbox but not on my machine. Show me how it can access an unmounted directory.

> Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy.

Amazing-sandbox does not solve for that. The current risk is contamination; if you are running `trivy`, it should not need access to tokens in a different env/directory.

> All of these problems and many more go away if the project is built around them from the start.

Please elaborate on your approach that will all me to run markdown/JS/Python/Go/Rust linters and security scanners. Remember that `trivy` which caused `litellm` compromise is a security scanner itself.

> developers can write sandboxes for their software more effectively because they can architect around the sandbox,

Yeah, let's ask 100+ linter providers to write sandboxes for you. I can't even get maintainers to respond to legitimate & trivial PRs many a time.

So it says right on the tin of my favorite distro: 'Warning: Beware that the docker group membership is effectively equivalent to being root! Consider using rootless mode below.' So # docker run super-evil-oci-container with a bind mount or two and your would-be attacker doesn't need to guess your sudo password.

1 - https://github.com/ashishb/amazing-sandbox

Same logic for software developers.

My dad was confident he could figure it out based on his perplexity Pro account. He attacked the problem from several angles and used it for help with what to do, how to do it, what to ask for when visiting offices, how to press them to move forward, and tons of other things.

Got the problem resolved.

So it definitely can reduce hiring lawyers even.

Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?

Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.

It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.

Obviously read it before running it etc.

the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion

Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518

also how are we sure that docker images aren't affected?

That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.

EDIT: no, it's compromised, see proxy/proxy_server.py.

Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)

Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)

I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

We have deleted all our pypi publishing tokens.

Our accounts had 2fa, so it's a bad token here.

We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).

https://socket.dev/blog/trivy-under-attack-again-github-acti...

https://github.com/BerriAI/litellm/issues/24518#issuecomment...

In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

Privilege escalation on macOS is very hard indeed. Apple have been improving security for a long time, it is far, far ahead of Linux or Windows in this regard. The default experience in Xcode is that a release-mode app you make will be sandboxed, undebuggable, have protected keychain entries other apps can't read, have a protected file space other apps can't read, and its own code will also be read-only to other apps. So apps can't interfere with each other or escalate to each other's privileges even when running as the same UNIX user. And that's the default, you don't have to do anything to get that level of protection.

Token in CI could've been way too broad.

From a security standpoint, you would rather pull in a library that is compromised and run a credential stealer? It seems like this is the exact intended and best behavior.

More strangely (to me), this is often addressed by adding loads of fallible/partial caching (in e.g. CICD or deployment infrastructure) for package managers rather than building and publishing temporary/per-user/per-feature ephemeral packages for dev/testing to an internal registry. Since the latter's usually less complex and more reliable, it's odd that it's so rarely practiced.

No one initially knows how much is compromised

That said, I'm sorry this is being downvoted: it's unhappily observing facts, not arguing for a different security response. I know that's toeing the rules line, but I think it's important to observe.

i'd much rather see a million open PRs than a single malicious PR sneak through due to lack of thorough review.

Which sounds great, but the way things work now tend to be the exact opposite of that, so there will be no trustable platform to run the untrusted code in. If the sandbox, or the operating system the sandbox runs in, will get breaking changes and force everyone to always be on a recent release (or worse, track main branch) then that will still be a huge supply chain risk in itself.

https://wiki.debian.org/SecureBoot#Shim

  > This assumes that we can get a locked down, secure, stable bedrock system and sandbox that basically never changes except for tiny security updates that can be carefully inspected by many independent parties.

Don't just update every time the dependency project updates. Update specifically for security issues, new features, and specific performance benefits. And even then avoid the latest version when possible.

To really run code without trust would need something more like a microkernel that is the only thing in my system I have to trust, and everything running on top of that is forced to behave and isolated from everything else. Ideally a kernel so small and popular and rarely modified that it can be well tested and trusted.

How do you change developer and user habits though? It's not as easy as people think.

https://bootstrappable.org/ https://github.com/crev-dev/

Not really. You should limit the attack surface for third-party code.

A linter running in `dir1` should not access anything outside `dir1`.

This is the problem with software progressivism. Some things really should just be what they are, you fix bugs and security issues and you don't constantly add features. Instead everyone is trying to make everything have every feature. Constantly fiddling around in the guts of stuff and constantly adding new bugs and security problems.

Software people could (mostly) trust each other's OSS contributions because we could trust the discipline it took in the first place. Not any more.

I would think humans have been using psychedelics since before we figured out meditation. Likely even before we were humans.

I also like the drunk monkey hypothesis.

No it's not. The blame game was very popular in the Eastern Block and it resulted in a stagnant society where lots of things went wrong anyway. For instance, Chernobyl.

Who's gonna enforce that?

> If you're compromised, you're burned forever in the ledger.

Guess we can't use XZ utils anymore cause Lasse Collin got pwned.

Also can't use Chalk, debug, ansi-styles, strip-ansi, supports-color, color-convert and others due to Josh Junon also ending up a victim.

Same with ua-parser-js and Faisal Salman.

Same with event-stream and Dominic Tarr.

Same with the 2018 ESLint hack.

Same with everyone affected by Shai-Hulud.

Hell, at that point some might go out of their way to get people they don't like burned.

At the same time, I think that stopping reliance on package managers that move fast and break things and instead making OS maintainers review every package and include them in distros would make more sense. Of course, that might also be absolutely insane (that's how you get an ecosystem that's from 2 months to 2 years behind the upstream packages) and take 10x more work, but with all of these compromises, I'd probably take that and old packages with security patches, instead of pulling random shit with npm or pip or whatever.

Though having some sort of a ledger of bad actors (instead of people who just fuck up) might also be nice, if a bit impossible to create - because in the current day world that's potentially every person that you don't know and can't validate is actually sending you patches (instead of someone impersonating them), or anyone with motivations that aren't clear to you, especially in the case of various "helpful" Jia Tans.

This bit sounds like dystopian governance, antithetical to most open source philosophies.

I don't blame the anons but the people blindly pulling in anon dependencies. The anons don't owe us anything.

If someone wants a package manager where all projects mandate verifiable ID that's fine, but I don't see that getting many contributors. And I also don't see that stopping people using fraudulent IDs.

I see people going in the opposite direction with "dump everything in front of my army of LLMs" setups. Horribly insecure, but gotta go fast, right?

https://github.com/calebfaruki/tightbeam https://github.com/calebfaruki/airlock

This is literally the thing I'm trying to protect against.

Otherwise people will naysay and detract from the cause. "It worked before" they will say. "Why don't we do it like before?"

DISA STIG already forbids use of the EPEL for Red Hat Enterprise Linux. Enterprise software install instructions are littered with commands to turn off gpgcheck and install rpm's from sourceforge. The times are changing and we need cryptographically verifiable guarantees of safety!

The problems you mentioned resonated a lot with me and why I'm building it, any interest in working to solve that together?: https://github.com/smol-machines/smolvm

- a security focused project should NOT default to train people installing by piping to bash. If i try previewing the install script in the browser it forces download instead of showing as plain text. The first thing i see is an argument

# --prefix DIR Install to DIR (default: ~/.smolvm)

that later in the script is rm -rf deleting a lib folder. So if i accidentally pick a folder with ANY lib folder this will be deleted.

- Im not sure what the comparison to colima with krunkit machines is except you don't use vm images but how this works or how it is better is not 100% clear

- Just a minor thing but people don't have much attention and i just saw aws and fly.io in the description and nearly closed the project. it needs to be simpler to see this is a local sandbox with libkrun NOT a wrapper for a remote sandbox like so many of the projects out there.

Will try reaching you on some channel, would love to collaborate especially on devX, i would be very interested in something more reliable and bit more lightweight in placce of colima when libkrun can fully replace vz

1. In comparison with colima with krunkit, I ship smolvm with custom built kernel + rootfs, with a focus on the virtual machine as opposed to running containers (though I enable running containers inside it).

The customizations are also opensource here: https://github.com/smol-machines/libkrunfw

2. Good call on that description!

I've reached out to you on linkedin

One should never trust the binaries, always build them from source, all the way down to the bootloader.

https://bootstrappable.org/

Checking all the files is really the only way to deal with potential malware, or even security vulns.

https://github.com/crev-dev/

The .tar.gz can be checksummed and saved (to be sure later on that you install the same .tar.gz and to be sure it's still got the same checksum). Piping to Bash in one go not so much. Once you intercept the .tar.gz, you can both reproduce the exploit if there's any (it's too late for the exploit to hide: you've got the .tar.gz and you may have saved it already to an append-only system, for example) and you can verify the checksum of the .tar.gz with other people.

The point of doing all these verifications is not only to not get an exploit: it's also to be able to reproduce an exploit if there's one.

There's a reason, say, packages in Debian are nearly all both reproducible and signed.

And there's a reason they're not shipped with piping to bash.

Other projects shall offer an install script that downloads a file but verifies its checksum. That's the case of the Clojure installer for example: if verifies the .jar. Now I know what you're going to say: "but the .jar could be backdoored if the site got hacked, for both the checksum in the script and the .jar could have been modified". Yes. But it's also signed with GPG. And I do religiously verify that the "file inside the script" does have a valid signature when it has one. And if suddenly the signing key changed, this rings alarms bells.

Why settle for the lowest common denominator security-wise? Because Anthropic (I pay my subscription btw) gives a very bad example and relies entirety on the security of its website and pipes to Bash? This is high-level suckage. A company should know better and should sign the files it ships and not encourage lame practices.

Once again: all these projects that suck security-wise are systematically built on the shoulders of giants (like Debian) who know what they're doing and who are taking security seriously.

This "malware exists so piping to bash is cromulent" mindset really needs to die. That mentality is the reason we get major security exploits daily.

It can be dedicated to a single service (or a full OS), runs a real BSD kernel, and provides strong isolation.

Overall, it fits into the "VM is the new container" vision.

Disclaimer: I'm following iMil through his twitch streams (the developer of smolBSD and a contributor to NetBSD) and I truly love what he his doing. I haven't actually used smolBSD in production myself since I don't have a need for it (but I participated in his live streams by installing and running his previews), and my answer might be somewhat off-topic.

More here <https://hn.algolia.com/?q=smolbsd>

At a glance, it's a matter of compatibility, most software has first class support for linux. But very interesting work and I'm going to follow it closely

worked in AWS and specifically with firecracker in the container space for 4 years - we had a very long onboarding doc to dev on firecracker for containers... So I made sure to focus on ease of use here.

The harder problem is that CI pipelines routinely grant scanner processes more credential access than they need. Trivy needed read access to the repo and container layers; it didn't need PyPI publish tokens. Scoping CI secrets to the minimum necessary operation, and injecting them only for the specific job that needs them rather than the entire pipeline, would have contained the blast radius here.

I haven't kept up with the recent exploits, so a side question: Have any of the recent supply chain attacks or related exploits included any escapes from basic dev containers?

We should have sandboxing in Rust, Python, and every language in between.

u can run in chroot jail and it wouldnt have accessed ssh keys outside of the jail...

theres many more similar technologies aleady existing, for decades.

doing it on a per language basis is not ideal. any new language would have to reinvent the wheel.

better to do it at system level. with the already existing tooling.

openbsd has plege/unveil, linux chroot, namespaces, cgroups, freebsd capsicum or w/e. theres many of these things.

(i am not sure how well they play within these scenarios, but just triggering on the sandboxing comment. theres plenty of ways to do it as far as i can tell...)

The problem is that programs can be entire systems, so "doing it at the system level" still means that you'd have to build boundaries inside a program.

you can use OS apis to isolate the thing u want to use just fine..

and yes, if you mix privilege levels in a program by design then u will have to design your program for that.

this is simple logic.

a programming language can not decide for you who and what you trust.

For the sake of the argument, what if I wanted to isolate numpy from scipy?

Would you run numpy in a separate process from scipy? How would you share data between them?

Yes, you __can__ do all of that without programming language support. However, language support can make it much easier.

https://en.wikipedia.org/wiki/Object-capability_model

Happily sandboxing almost all third-party tools since 2025. `npm run dev` does not need access to my full disk.

This is the interesting part. What kind of UI or other mechanisms would help here? There's no silver bullet for detecting and crashing on "something bad". The adversary can test against your sandbox as well.

Making this work on a per-library level … seems a lot harder. The cost for being very paranoid is a lot of processes right now.

(Yes, they should better configure which CI job has which permissions, but this should be the default or it won't always happen)

Then we talk about containment like anyone actually looked at that dep list.

No free lunch. LLMs are capable of writing exploitable code and you don’t get notifications (in the eg Dependabot sense, though it has its own problems) without audits.

But mention that on HN and watch getting downvoted into oblivion: the war against general computation, walled gardens, locked down against device owners...

I updated my global configs to set min release age to 7 days:

  ~/.config/uv/uv.toml
  exclude-newer = "7 days"
  
  ~/.npmrc
  min-release-age=7 # days
  
  ~/Library/Preferences/pnpm/rc
  minimum-release-age=10080 # minutes
  
  ~/.bunfig.toml
  [install]
  minimumReleaseAge = 604800 # seconds

eg:

  npm install <package> --min-release-age 0
  
  pnpm add <package> --minimum-release-age 0
  
  uv add <package> --exclude-newer "0 days"
  
  bun add <package> --minimum-release-age 0

  $ pnpm add -D typescript@6.0.2
   ERR_PNPM_NO_MATURE_MATCHING_VERSION  No matching version found for typescript@6.0.2 published by Wed Mar 18 2026..

  pnpm config set minimumReleaseAge 10080 --global

https://pnpm.io/cli/config

https://ramimac.me/trivy-teampcp/#phase-09

The investment compounds! I have enough context to quickly vet incoming information, then it's trivial to update a static site with a new blurb

I made this tool for macos systems that helps detect when a package accesses something it shouldn't. it's a tiny go binary (less than 2k LOC) with no dependencies that will mount a webdav filesystem (no root) or NFS (root required) with fake secrets and send you a notification when anything accesses it. Very stupid simple. I've always really liked the canary/honeypot approach and this at least may give some folks a chance to detect (similar to like LittleSnitch) when something strange is going on!

Next time the attack may not have an obvious performance issue!

I always wanted to mess with building virtual filesystems but was unwilling to venture outside the standard library (i.e. libfuse) for reasons wonderfully illustrated in this thread and elsewhere. Somehow the idea of implementing a networked fs protocol and leaving system integration to the system never crossed my mind.

I'm glad more people are taking this stance. Large centralized standard libraries and minimal audited dependencies is really the only way to achieve some semblance of security. There is simply no other viable approach.

Edit: What's the license for this project?

I would expect better spam detection system from GitHub. This is hardly acceptable.

I scrolled through and clicked a few profiles. While many might be spam accounts or low-activity accounts, some appeared to be actual GitHub users with a history of contributions.

I’m curious how so many accounts got compromised. Are those past hacks, or is this credential steeling hack very widespread?

Are the trivy and litellm hacks just 2 high profile repos out of a much more widespread “infect as many devs as possible, someone might control a valuable GitHub repository” hack? I’m concerned that this is only the start of many supply chain issues.

Edit: Looking through and several of the accounts have a recent commit "Update workflow configuration" where they are placing a credential stealer into a CI workflow. The commits are all back in february.

Worked like a charm, much appreciated.

This was the answer I was looking for.

Thanks, that helped!

Thanks for the tip!

Great explanation, thanks for sharing.

This was the answer I was looking for.

It could be solved by the filter but filter would also have a bunch of false positives

It just doesn't have to be spammed enough that advertisers leave the platform and I think that they sort of succeed in doing so.

Think about it, if Facebook shows you AI slop ragebait or any rage-inducing comment from multiple bots designed to farm attention/for malicious purposes in general, and you fall for it and show engagement to it on which it can show you ads, do you think it has incentive to take a stance against such form of spam

I'm not sure that's actually true. It's just that at scale this is still a hard problem that you don't "just" fix by running a simple filter as there will be real people / paying customers getting caught up in the filter and then complain.

Having "high engagement" doesn't really help you if you are optimizing for advertising revenue, bots don't buy things so if your system is clogged up by fake traffic and engagement and ads don't reach the right target group that's just a waste.

Even still though, we can't really trust any open-source software any more that has third party dependencies, because the chains can be so complex and long it's impossible to vet everything.

It's just too easy to spam out open-source software now, which also means it's too easy to create thousands of infected repos with sophisticated and clever supply chain attacks planted deeply inside them. Ones that can be surfaced at any time, too. LLMs have compounded this risk 100x.

This is why software written in Rust scares me. Almost all Rust programs have such deep dependency trees that you really can't vet them all. The Rust and Node ecosystems are the worst for this, but Python isn't much better. IMO it's language-specific package managers that end up causing this problem because they make it too easy to bring in dependencies. In languages like C or C++ that traditionally have used system package managers the cost of adding a dependency is high enough that you really avoid dependencies unless they're truly necessary.

JS/TS > Screems aloud! never do "npm import [package containing entire world as dependency]"

Rust > Just import everything since rust fixes everything.

When you design your package management and doctrine like de facto javascript your have failed like javascript.

However, the broader idea of supply chain attacks remains challenging and AI doesn’t really matter in terms of how you should treat it. For example, the xz-utils back door in the build system to attack OpenSSH on many popular distros that patched it to depend on systemd predates AI and that’s just the attack we know about because it was caught. Maybe AI helps with scale of such attacks but I haven’t heard anyone propose any kind of solution that would actually improve reliability and robustness of everything.

[1] Fully Countering Trusting Trust through Diverse Double-Compiling https://arxiv.org/abs/1004.5534

TLDR: what I said is only foolish if you take the absolute dumbest possible interpretation of it.

You're right though. There's been talks of a big global hack attack for a while now.

Nothing is safe anymore. Keep everything private airgapped is the only way forward. But most of our private and personal data is in the cloud, and we have no control over it or the backups that these companies keep.

While LLMs unlock the opportunity to self-host and self-create your infrastructure, it also unleashes the world of pain that is coming our way.

The kernel is not just open source, it's a very fast-moving codebase. That's how we win all wars against AI-authored exploits. While the LLM trains on our internal APIs, we change the APIs — by hand. When the agent finally submits its pull request, it gets lost in unfamiliar header files and falls into a state of complete non-compilability. That is the point. That is our strategy.

1 - https://en.wikipedia.org/wiki/Demobbed_(2000_film)

In the end we need fully deterministic, 100% verifiable, chains. From the tiny boostrapped beginning, to the final thing.

There are people working on these things. Both, in a way, "top-down" (bootstrapping a tiny compiler from a few hundred bytes) and "bottom-up" (a distro like Debian having 93% of all its packages being fully reproducible).

While most people are happy saying "there's nothing wrong with piping curl to bash", there are others that do understand what trusting trust is.

As a sidenote although not a kernel backdoor, Jia Tan's XZ backdoor in that rube-goldberg systemd "we modify your SSHD because we're systemd and so now SSHD's attack surface is immensely bigger" was a wake-up call.

And, sadly and scarily, that's only for one we know about.

I think we'll see much more of these cascading supply chains attack. I also think that, in the end, more people are going to realize that there are better ways to both design, build and ship software.

The possibilities within a good threat could be catastrophic if we assume so, and if we assume nation-states to be interested in sponsoring hacking attacks (which many nations already do) to attack enemy nations/gain leverage. We are looking at damage within Trillions at that point.

But I would assume that Linux might be safe for now, it might be the most looked at code and its definitely something safe.

LLVM might be a bit more interesting as it might go a little unnoticed but hopefully people who are working at LLVM are well funded/have enough funding to take a look at everything carefully to not have such a slip up.

I've been through SOC2 certifications in a few jobs and I'm not sure it makes you bullet proof, although maybe there's something I'm missing?

But all it means in the end is you can read up on how a company works and have some level of trust that they're not lying (too much).

It makes absolutely zero guarantees about security practices, unless the documented process make these guarantees.

I mean ideally a proper SOC2 would mean there are processes in place to reduce the likelihood of this happening, and then also processes to recover from if it did ended up happening.

But the end result could've been essentially the same.

https://news.ycombinator.com/item?id=47481729

Basically it forkbombed `grep -r rpcuser\rpcpassword` processes trying to find cryptowallets or something. I saw that they spawned from harness, and killed it.

Got lucky, no backdoor installed here from what i could make out of the binary

how do you do that? have Activity Monitor up at all times?

This threat actor seems to be very quickly capitalising on stolen credentials, wouldn’t be surprised if they’re leveraging LLMs to do the bulk of the work.

https://news.ycombinator.com/item?id=47475888

Do the labs label code versions with an associated CVE to label them as compromised (telling the model what NOT to do)? Do they do adversarial RL environments to teach what's good/bad? I'm very curious since it's inevitable some pwned code ends up as training data no matter what.

I assume most labs don't do anything to deal with this, and just hope that it gets trained out because better code should be better rewarded in theory?

Since they all seem positive, it doesn't seem like an attack but I thought the general etiquette for github issues was to use the emoji reactions to show support so the comment thread only contains substantive comments.

> It also seems that attacker is trying to stifle the discussion by spamming this with hundreds of comments. I recommend talking on hackernews if that might be the case.

They also seem to be spilling into HN [1].

Runaway AI agents? A meme I'm to old to understand?

[1] https://news.ycombinator.com/item?id=47508315

[1] https://ramimac.me/teampcp/#spam-flood-litellm [2] https://ramimac.me/teampcp/#discussion-flooded

Run all your new dependencies through static analysis and don't install the latest versions.

I implemented static analysis for Python that detects close to 90% of such injections.

https://github.com/rushter/hexora

1. pin dependencies with sha signatures 2. mirror your dependencies 3. only update when truly necessary 4. at first, run everything in a sandbox.

> ### Software Supply Chain is a Pain in the A*

> On top of that, the room for vulnerabilities and supply chain attacks has increased dramatically

AI Is not about fancy models, is about plain old Software Engineering. I strongly advised our team of "not-so-senior" devs to not use LiteLLM or LangChain or anything like that and just stick to `requests.post('...')".

[0] https://sb.thoughts.ar/posts/2025/12/03/ai-is-all-about-soft...

Configure the CI to make a release with the artefacts attached. Then have an entirely private repo that can't be triggered automatically as the publisher. The publisher repo fetches the artefacts and does the pypi/npm/whatever release.

https://docs.npmjs.com/generating-provenance-statements

https://packaging.python.org/en/latest/specifications/index-...

some will even audit each package in there (kind crap job but it works fairly well as mitigation)

LiteLLM wouldn't be my top choice, because it installs a lot of extra stuff. https://news.ycombinator.com/item?id=43646438 But it's quite popular.

We are looking at similar attack vectors (pth injection), signatures etc. in other PyPI packages that we know of.

or pyproject.toml (not possible to filter based on absence of a uv.lock, but at a glance it's missing from many of these): https://github.com/search?q=path%3A*%2Fpyproject.toml+%22%5C...

or setup.py: https://github.com/search?q=path%3A*%2Fsetup.py+%22%5C%22lit...

It would be interesting if Python, NPM, Rubygems, etc all just decided to initiate an ecosystem-wide credential reset. On one hand, it would be highly disruptive. On the other hand, it would probably stop the damage from spreading.

https://inspector.pypi.io/project/litellm/1.82.8/packages/fd...

I got popped by our security team, they were convinced I had this malware because my machine attempted to connect to the checkmarx domain.

clearly a false positive but I still had to roll credentials and wipe my machine.

This stupidity is squarely on GitHub CI. Trivy is also bad here but the blast radius should have been more limited.

https://github.com/crewAIInc/crewAI/commit/8d1edd5d65c462c3d...

One thing not in that writeup is that very little action was needed for my engineer to get pwnd. uvx automatically pulled latest litellm (version unpinned) and built the environment. Then Cursor started up the local MCP server automatically on load.

Now I am not worried about the Ai Api keys having much damage but I am thinking of one step further and I am not sure how many of these corporations follow privacy policy and so perhaps someone more experienced can tell me but wouldn't these applications keep logs for legal purposes and those logs can contain sensitive information, both of businesses but also, private individuals perhaps too?

Irrevocable transfers... What could go wrong?

But, one of the arguments that I saw online from this was that when a security researcher finds a bug and reports it to the OSS project/Company they then fix the code silently and include it within the new version and after some time, they make the information public

So if you run infrequently updated versions, then you run a risk of allowing hackers access as well.

(An good example I can think of is OpenCode which had an issue which could allow RCE and the security researcher team asked Opencode secretly but no response came so after sometime of no response, they released the knowledge in public and Opencode quickly made a patch to fix that issue but if you were running the older code, you would've been vulnerable to RCE)

How you use it depends on your workflow. An entry like this in your pyproject.toml could suffice:

  [tool.uv]
  exclude-newer = "5 days"

The package was directly compromised, not “by supply chain attack”.

If you use the compromised package, your supply chain is compromised.

it does a lot of CPU intensive work

    spawn background python
    decode embedded stage
    run inner collector
    if data collected:
        write attacker public key
        generate random AES key
        encrypt stolen data with AES
        encrypt AES key with attacker RSA pubkey
        tar both encrypted files
        POST archive to remote host

Also the repo is so active that it's very hard to understand the state of issues and PRs, and the 'day 0' support for GPT-5.4-nano took over a week! Still, tough situation for the maintainers who got hacked.

Oh boy supply chain integrity will be an agent governenace problem, not just a devops one. If you send out an agent that can autonomously pull packages, do code, or access creds, then the blast radius of compromises widens. That's why I think there's an argument for least-privilege by default--agents should have scoped, auditable authority over what they can install and execute, and approval for anything outside the boundaries.

[1]: https://pypi.org/project/litellm/#history

I guess I am lucky as I have watchtower automatically update all my containers to the latest image every morning if there are new versions.

I also just added it to my homelab this sunday, I guess that's good timing haha.

https://github.com/zizmorcore/zizmor

EDIT: here's what I did, would appreciate some sanity checking from someone who's more familiar with Python than I am, it's not my language of choice.

find / -name "litellm_init.pth" -type f 2>/dev/null

find / -path '/litellm-1.82..dist-info/METADATA' -exec grep -l 'Version: 1.82.[78]' {} \; 2>/dev/null

  rg litellm  --iglob='*.lock'

no i don't let it connect to web...

Basically, have all releases require multi-factor auth from more than one person before they go live.

A single person being compromised either technically, or by being hit on the head with a wrench, should not be able to release something malicious that effects so many people.

Though, the secondary doesn't necessarily have to be a maintainer or even a contributor on the project. It just needs to be someone else to do a sanity check, to make sure it is an actual release.

Heck, I would even say that as the project grows in popularity, the amount of people required to approve a release should go up.

How do I even know who to trust, and what prevents two people from conspiring together with a long con? Sounds great on the surface but I'm not sure you've thought it through.

As far as who to trust, I could imagine the maintainers of different high-level projects helping each other out in this way.

Though, if you really must allow a single user to publish releases to the masses using existing shared social infrastructure. Then you could mitigate this type of attack by adding in a time delay, with the ability for users to flag. So instead of immediately going live, add in a release date, maybe even force them to mention the release date on an external system as well. The downside with that approach is that it would limit the ability to push out fixes as well.

But I think I am OK with saying if you're a solo developer, you need to bring someone else on board or host your builds yourself.

Better to enforce good security standards than cripple the ecosystem.

There is also the need for data sanitation, because the attacker could distribute compromised files through user’s data which will later be run and compromise the host.

1 - https://github.com/ashishb/amazing-sandbox

https://greyhaven.co/insights/how-greywall-prevents-every-st...

[0] https://greywall.io/

https://github.com/Nayjest/lm-proxy

``` pip install lm-proxy ```

Guys, sorry, as the author of a competing opensource product, I couldn’t resist

Domains might get added to a list for things like 1.1.1.2 but as you can imagine that has much smaller coverage, not everyone uses something like this in their DNS infra.

https://github.com/krrishdholakia/blockchain/commit/556f2db3...

  - # blockchain
  - Implements a skeleton framework of how to mine using blockchain, including the consensus algorithms.
  + teampcp owns BerriAI

This would also disable site import so not viable generically for everyone without testing.

As described in https://docs.python.org/3/library/site.html :

> Lines starting with import (followed by space or tab) are executed.... The primary intended purpose of executable lines is to make the corresponding module(s) importable (load 3rd-party import hooks, adjust PATH etc).

So what malware can do is put something in a .pth file like

  import sys;exec("evil stringified payload")

  import_=exec("evil stringified payload")

.pth files have worked this way since 2.1 (comparing https://docs.python.org/2.1/lib/module-site.html to https://docs.python.org/2.0/lib/module-site.html). As far as I can tell there was no PEP for that change.

if you have tips i am sure they are welcome. snark remarks are useless. dont be a sourpuss. if you know better, help the remediation effort.

[1]: https://pypi.org/project/litellm/

This was taught in the 90s. Sad to see that lesson fading away.

First Trivy (which got compromised twice), now LiteLLM.

An actual infosec audit would have rigorously enforced basic security best practices in preventing this supply chain attack.

[0] https://news.ycombinator.com/item?id=47502754

The Python ecosystem provides too many nooks and crannies for malware to hide in.

I hope that everyone's course of action will be uninstalling this package permanently, and avoiding the installation of packages similar to this.

In order to reduce supply chain risk not only does a vendor (even if gratis and OS) need to be evaluated, but the advantage it provides.

Exposing yourself to supply chain risk for an HTTP server dependency is natural. But exposing yourself for is-odd, or whatever this is, is not worth it.

Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

And even if you weren't using this specific dependency, check your deps, you might have shit like this in your requirements.txt and was merely saved by chance.

An additional note is that the dev will probably post a post-mortem, what was learned, how it was fixed, maybe downplay the thing. Ignore that, the only reasonable step after this is closing a repo, but there's no incentive to do that.

Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.

[0] https://platform.claude.com/docs/en/api/openai-sdk

That's what they pay us for

I'd get it if it were a hassle that could be avoided, but it feels like you are trying to avoid the very work you are being paid for, like if a MCD employee tried to pay a kid with Happy Meal toys to work the burger stand.

Another red flag, although a bit more arguable, is that by 'abstracting' the api into a more generic one, you achieving vendor neutrality, yes, but you also integrate much more loosely with your vendors, possibly loose unique features (or can only access them with even more 'hassle' custom options, and strategically, your end product will veer into commodity territory, which is not a place you usually want to be.

This is like a couple hours of work even without vibe coding tools.

There's more than that (even if most other systems also provide a OpenAI compatible API which may or may not expose either all features of the platform or all features of the OpenAI API), and the differences are not cosmetic, but since LiteLLM itself just presents an OpenAI-compatible API, it can't be providing acccess to other vendor features that don't map cleanly to that API, and I don't think its likely to be using the native API for each and being more complete in its OpenAI-compatible implementation of even the features that map naturally than the first-party OpenAI-compatibility APIs.)

Back in the days of GPT-3.5, LiteLLM was one of the projects that helped provide a reliable adapter for projects to communicate across AI labs' APIs and when things drifted ever so slightly despite being an "OpenAI-compatible API", LiteLLM made it much easier for developers to use it rather than reinventing and debugging such nuances.

Nowadays, that gateway of theirs isn't also just a funnel for centralizing API calls but it also serves other purposes, like putting guardrails consistently across all connections, tracking key spend on tokens, dispensing keys without having to do so on the main platforms, etc.

There's also more to just LiteLLM being an inference gateway too, it's also a package used by other projects. If you had a project that needed to support multiple endpoints as fallback, there's a chance LiteLLM's empowering that.

Hence, supply chain attack. The GitHub issue literally has mentions all over other projects because they're urged to pin to safe versions since they rely on it.

Hundreds of downvoted comments like "Worked like a charm, much appreciated.", "Thanks, that helped!", and "Great explanation, thanks for sharing."

It's pretty disappointing that safetensors has existed for multiple years now but people are still distributing pth files. Yes it requires more code to handle the loading and saving of models, but you'd think it would be worth it to avoid situations like this.

   # add any dependency file patterns
   osv-scanner -r .

https://github.com/BerriAI/litellm/issues/24512#issuecomment...

The previous version triggers on `import litellm.proxy`

Again, all according to the issue OP.

[1] https://docs.python.org/3/library/site.html

I'm sensing a pattern here, hmm.

https://news.ycombinator.com/newsguidelines.html#generated

https://www.youtube.com/watch?v=aoag03mSuXQ