Okay, checked, it was March of this year: https://futuresearch.ai/blog/litellm-pypi-supply-chain-attac.... What even is time anymore in the llm age?

At one point permission pop-up fatigue sets in we start giving any permission as if it's normal. We also become desensitised to isntalling tools: brew, npm, it becomes second nature.

"Oh, those 367 tools installed by package X? Yeah yeah, must be normal, that's how all these work now".

"Brew shows 59 packages outdated?". "Must upgrade". "Yeah, done. Perfect".

And the advent of agents and harnesses. It's not a matter of whether, it's when! OSes need to do something about it. Help users have complete separate "program space" at ease with better, easier, and lighter VMs if nothing else. I know, I know - Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies. Of course.

LiteLLM is frequently used in roughly the same scenario - trading virtual LAN-only keys for real ones as it relays requests to the inference providers.

LiteLLM isn't a sandboxing tool so it just can't perform as much risk reduction even when everything is working as intended.

And what nono adds over sandbox-exec is a uniform interface across platforms and some policy management tools. You can impose the same restrictions via the OS built-in mechanisms yourself via Seatbelt or a built-in interface like sandbox-exec. You can also combine an OS-level sandboxing tool like nono, zerobox, Anthropic's srt, Seatbelt, landrun, Bubblewrap, Fence, etc., with a VM-based one like Docker's sbx, smolvm, krunvm, microsandbox, or Shuru. So now you need two simultaneous supply chain attacks in unrelated projects to line up just right at the same time.

"Eventually you have to trust some part of your security stack" is neither particularly interesting nor evidence that there's no way to defend against supply chain attacks. Even if you only have one sandboxing tool and it's OS-level rather than VM-based, that becomes a single part of your stack that you can give special attention to in evaluating updates in a way that LiteLLM can't be.

Sandbox comes in for the unknown ones and yes, its a pain to get the profile right.

From some of them perhaps, but plenty hide for -years- undetected.

I mean, didn't Linux shells invent pop-up fatigue? I thought we (yes, we) all used to -y flag preemptively so the upgrades and installations/setups go through. Linux really did do it first, the agents are just more needy and their popups more verbose now, but not exactly new.

VM isn't a silver bullet.

Either way it's got nothing to do with security - I was never going to catch a vulnerability in the central debian repos by reading a list of dependent packages.

> No public commit in BerriAI/litellm contains data = "cook" or a cook/ folder.

> SafeDep assesses, with moderate to high confidence, that BerriAI’s private release pipeline built these wheels from an attacker-modified git source.

They revoked all but one developers, which is probably the right approach, but I'd pin LiteLLM to "something before this version" and keep it there until further notice!

All I read was that it creates a directory from the build file but seemingly no description how new files or added dependencies slipped by. Was `cook.pth` a dependency that got automatically downloaded from PyPI on install and then the build system put it into dependencies?

--

Leaving this for others:

> BerriAI’s private release pipeline built these wheels from an attacker-modified git source. The attacker could have used a compromised maintainer or bot credential, a private branch or mirror, or the reference the pipeline builds from. A stolen PyPI token used from the attacker’s own machines fits the evidence less well.

Makes sense. It was that a maintainer or the private pipeline was attacked and the malicious dependency was injected there by squatting on the maintainer's local copy rather than the dependency being hijacked for everyone. Very clever.

What would catch it is someone rebuilding from the public tag and comparing the result, which almost nobody does for the python wheels. I have been treating attestations as the answer for my own project's releases, so this is a good reminder of where they stop.

I’m unsure how to automate this properly, maybe an LLM is the best starting point.

Ultimately, you need someone to verify releases and that probably means paying for it. I know Sonatype paid solutions utilise their own internal research to allow/block new releases.

Risk is ensuring enough security researchers are finding these and quarantining these before that default cooldown lapses.

I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.

I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.

If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time

That said, I am looking into alternatives. Two times in one year is really a terrible look.

Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.

I don't think memory safety would have helped here.

> the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker

You mean the Rust ecosystem?

Taking the definition from another user on this comment-chain:

"an OpenAI compatible endpoint routing across multiple providers based on latency"

If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.

- LLM_HOST = "api.openai.com" + LLM_HOST = "api.competitor.com"

It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.

I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.

second? i haven't kept count, but we have to be somewhere in the hundreds of chances now

Turns out Ancient Aliens was onto something all along.

> OSS Users > For the time being, please pin your version to 1.106.2 or any explicit version published prior to now.

> Enterprise Customers > There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27. Our private registry is not affected. If you are an enterprise customer pulling from our private registry you are not affected.

Seems a little unethical

1. there is a second vulnerability that will be disclosed next Tuesday (can't be this one because talking about it as undisclosed in the GH issue disclosing it makes no sense)

2. the patch for that vulnerability is somewhere in the diff between $current_public_release and $current_enterprise_release

3. You have to pay to get $current_enterprise_release

They might not be selling the vulnerability directly, but they are selling access to the patch that fixes it. Which would be enough of a hint even in pre-LLM times. And unlike other vendors they don't just claim "you get faster updates and additional features" or something like that, they directly point out that if you buy now you get advance access to the fix to an unannounced vuln

I'm not saying they are advertising this to bad actors. But if they wanted to do that with a shred of plausible deniability, this would be the way to do it

There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.

We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.

However, if anyone is looking for 20x faster and less "targeted" LiteLLM alternative - I'm work on one in Go. That's why GoModel it's called. https://github.com/ENTERPILOT/GoModel/

I myself have written a small llm proxy in Go, doing nothing but injecting the real API key to outgoing requests while itself validating virtual keys it generates, and it works great.

- being resilient to upstream provider downtime

- centrally managed config/controls

- visibility and allocation into usage and costs

We are deploying a gateway at work primarily for these three.

At home, GoModel's virtual models have been super helpful for testing out new open weights. I can change the model deployed in vLLM without having to change any of my downstream tools, which get a stable name.

https://gomodel.enterpilot.io/docs/features/virtual-models

I use it, I don't cooldown, proper version pinning for me.

Tell em Tom Zubiri sent ya (I don't get a commission, just good karma)

Agents and malware will eradicate all goodwill left.

Good by internet.

/s

1792590232 | Litellm 1.107.0 Compromised | https://github.com/BerriAI/litellm/issues/47826 | https://news.ycombinator.com/item?id=50180845 | 0 comments

1. Google Cloud Outage [0]

2. eMudhra CA private key Leak [1]

3. PingCap TiDB 10.6 TB data Leak [2]

4. PHP2Shell uncovers critical CVEs (with AI) [3]

...and now we have Round 2 of a LiteLLM compromise after the first one in [4].

[0] https://news.ycombinator.com/item?id=50163961

[1] https://news.ycombinator.com/item?id=50166579

[2] https://news.ycombinator.com/item?id=50173716

[3] https://news.ycombinator.com/item?id=50180836

[4] https://news.ycombinator.com/item?id=47501426

https://github.com/BerriAI/litellm/issues/47826

oh boy. Announcing that there's a private enterprise repo you should be paying for if you don't want a credential stealer in your build is quite the optic, and then announcing there's an undisclosed second vulnerability that's being distributed, just not to the poors...

Also if you don't like organizations fixing vulns without releasing the open source first, you're going to be upset when you learn how every major vuln and associated patches are distributed long before the fixes hit the repos. A timely example of this is PHP, literally today.

Sorry, there was a paid-for PHP release that fixed it I could have gotten a week ago, and that they publicly announced existed, and it was just people who don't pay for PHP who had to wait...?

You can argue about whether the "publicly announced existed" part since they don't advertise for any single package, but that's the ecosystem you buy into.

The way you've written this is substantially misleading. RedHat receives notice of the vulnerabilities so they can prepare packages to be available the moment the embargo lifts, which I guess technically could hit the public packages a tiny bit earlier, but they are not releasing fixes of embargoed vulnerabilities to their commercial clients before the embargo lifts.

This is substantially different from the policy I've quoted that I am objecting to.

There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.

We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.

It's really not as bad as the accumulated headlines and exploitative hype make it sound.

It's similar to the mass hysteria and moral panics over things like child abduction. Media reports make certain dangers seem much more important than they really are, when considered in context.