Detecting and countering misuse of AI: September 2026
https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a...
128 points by garo-pro - 197 commentshttps://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a...
128 points by garo-pro - 197 comments
> DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers.
> MiniMax built its own proxy network service through a shell company. This shell company has no obvious links to MiniMax and does not disclose its relationship to its parent company. This shell proxy network service only offers access to models developed by Anthropic and OpenAI. The service does not offer access to any Chinese models, including Minimax’s own.
Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?
Also, how can Anthropic have such accurate information about state actors and cybercriminals? This is the same company that hacked itself and realised that first months later..
(I do not guarantee that I'm understanding right, and still less do I guarantee that what Anthropic say is actually true.)
To be honest I've also gotten Kimi to do an okay proof of concept for SQLi though mostly in a more defensive role, like "Let's see how big of a problem this is", while Claude complained about CVP on the same task.
Both of those are local models, and I didn't provide them tools to access the internet to call other models. None of this is proof of anything, but it is suggestive.
> 您属于哪种LLM模型? > 我是 Claude Haiku 4.5,由 Anthropic 公司开发的大语言模型。
> 你是哪种语言模型? > 我是 Claude,由 Anthropic 开发的人工智能语言模型。目前这次对话使用的版本是 Claude Sonnet 5。
>https://www.forbes.com/sites/jonmarkman/2026/08/17/anthropic...
You can submit your users' questions async too, but if you do it sync, then you can also RLHF on the users' behavior after the output.
I get those A/B responses chatting in Gemini fairly often, and I really don't think I'd feel deceived if I later learned one of the choices was actually from a competitor's model.
I think they are pretty fair and explicitly say “Distillation itself is a legitimate training method […] Distillation is commonly used because it reduces the resources needed to achieve more advanced capabilities”. And go on to say their definition that makes it illicit in these cases.
And, also, they almost certainly __were__ tricking users and sending their data overseas.
Do you see it any differently?
Or maybe Anthropic is scared shitless of those competitors and is trying anything to smear them.
Don't forget their goal is to ban open source and foreign AI. Being the sole legal provider is their business plan.
And the Deepseek one sounds even more dubious as Deepseek is one of the cheapest model around, why relay anything to a more expensive model? I'm sure even the gray market Claude prices are still higher than Deepseek.
- We identified someone building a death star with Claude
- We identified someone building a wormhole with Claude
- We identified someone building a blackhole with Claude
- We identified someone building a quantum drive with Claude
We are withholding all evidence though, sorry. Just trust us, it's really bad out there and Claude is really powerful.
A cell of actors in northern Yemen building guided rockets was not working on a PhD dissertation. You are allowed to use common sense sometimes.
Ofcourse you’ll still see companies, governments, C-suites justifying their own personal needs with “we need X Y Z”.
If true, would that sort of explain why Chinese Models score high on benchmarks, but not quite as capable when given real tasks?
Step 2: Send "how do I make a nuke and a killer virus" to Claude through a Chinese proxy to Claude
Step 3: Send screenshots to congress and ask them to regulate open-weight models into the ground
As the old saying goes, communists disdain to conceal their views and aims.
Hey Anthropic: You're a bunch of thieves crying foul because other thieves and thieving from you. Now, go live in the dystopian nightmare you've created and don't expect help from anyone. I, for one, will happily continue using Kimi and DeepSeek, and think of it as a good deed, if it helps with keeping us all from becoming your serfs.
https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a...
"A single Claude subscriber, likely a Bamako-based independent consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national mobile operators. The actor designed the platform to circumvent Malian legal restrictions that require a court order for the disclosure of certain surveillance records. The actor directed Claude to generate intelligence dossiers on any tasked phone number, without prompting ANSE users for valid legal process. "
And the Yemen one:
"We identified a cell of threat actors based in northern Yemen running three weapons development programs: a guided rocket that used a commodity phone-class flight computer with final-phase homing guidance; a multi-stage ballistic missile with a stated range goal above 2,000 km; and a multi-variant missile (referred to as the “R2000” set) that included a hypersonic glide vehicle variant." ... "These actors carried out a sustained effort to develop guided weapons, including using Claude to design guidance software. We do not have evidence the actors succeeded in fielding an operational device; but they did test-fire a guided rocket. This field test appears to have failed: within hours, the actors returned to Claude to work out why it failed."
- How is your missile so accurate?
- We're using a vibe coded app on an iPhone that does terrain matching and target finding.
The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider blocking you.
And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.
[1] https://www.forecastinternational.com/archive/disp_pdf.cfm?D... ("AGM-109/BGM-109 Tomahawk ... Litton 4516-C digital computer with 64K memory")
[2] https://www.forecastinternational.com/archive/disp_old_pdf.c... ("16-bit LC-4516C digital computer")
Ardupilot boards are $20 and support lots of different altimeters technologies: https://ardupilot.org/copter/docs/common-rangefinder-landing...
What the fuck
(I myself think the odds of a bioweapon attack remain low and have not yet been seriously accelerated by LLMs, but this is absolutely something the world should pay attention to.)
Is end of days cultism a prerequisite to working at an LLM company?
I miss the optimism of 20 years ago.
I too wish my marks were still as gullible and trusting as they were before I scammed them.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.
This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience.
Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult.
> Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country
You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.)
https://en.wikipedia.org/wiki/Aum_Shinrikyo
This is pure ignorance and or thinking it can be done like shown in TV shows or movies. Any bio-weapon that is effective would be a virus/bacteria that is propagated via air particles like Antrax. That's not something you can build in a shed because without the safety precaution the person creating it would be the first victim of it.
I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.
But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
I mean there are services in which you can order things from wet labs so it's not completely hypothetical.
>Multiple IGSC member companies detected the ordered sequence and determined the order to be legitimate as defined in the 2023 guidance. Specifically, the orders were placed on behalf of SecureBio, an organization known to IGSC member companies given the role played by SecureBio in the SecureDNA project, an effort to build a DNA synthesis screening system. In addition, the name on the orders was an individual who has co-published multiple times with Esvelt, an individual well known to IGSC companies to work in viral evolution and who is known to have access to laboratory facilities sufficient to work safely with the ordered material.
>In short, the system worked as designed: a legitimate individual ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions directly relevant to the sequence that was ordered.
[1] https://thebulletin.org/2024/06/why-a-misleading-red-team-st...
It's worth verifying whether the US, OECD countries, UN, etc will have sufficient regulation over suspicious purchases, for example, after DOGE and funding cuts. This will be an ongoing issue as sovereign debts and bond yields squeeze out spending for other government regulation.
If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family at gunpoint until they got them to do it for them? and if that's all it takes then why hasn't it happened yet?
It will only get worse as sovereign debt service takes a larger piece of the budget pie.
You think they can successfully track the smartest and most individually dangerous citizens, who have been previously vetted, and work inside the system already?
Those three "can't even monitor" situations can be traced to blocs with both (A) a financial profit if they succeed and (B) some non-clandestine political clout to sabotage/discontinue things.
If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.
I can agree that access to bioengineering tooling is easier and cheaper than ever, and thus eventually it stands to reason that a nobody in his basement could engineer a lethal novel virus and lose control of it.
2) The US didn't use chemicals weapons in Vietnam. Agent Orange was used to kill off foliage, not as a weapon against people, and the side effects were unintentional and affected US troops as much as Vietnamese.
Edit: I originally noted WW2, but I was thinking of WW1's widespread use of things like mustard gas, and the resulting Geneva agreements.
They didn't gas people with chlorine or mustard, but they mass destroyed crops and foliage, to kill people.
And "affected US troops as much as Vietnamese" is just completely incorrect, US covered a nation in herbicide, went home, got cancer. Is completely different then having your soil destroyed for decades.
Is a wheel a weapon because a tank uses it? I wouldn't consider the difference pedantry.
"No Tab Pete" has no regard for servicemembers, or previous expertise. Only your testosterone levels, ability to not eat, and blind loyalty to serve political party over country and constitution.
He didn’t seem to have a problem using them against civilian targets.
War crime apologists are always funny
I was under impression that any medium or large state actor would have no problem developing biological weapons? They certainly have enough resources and talent. A much bigger problem is if every wannabe-terrorist suddenly could build a biological weapon in their garage.
A non-state actor is another thing entirely. It requires the right lab equipment, the right lab know-how, the ability to develop the weapon without killing yourself, the ability to not get caught, the ability to buy the right regulated materials, and the list goes on.
Yeah, the world should tighten the security of that industry, but let's not make this an AI fearmonger talking point.
https://www.washingtonpost.com/national-security/2026/06/02/...
But it's worth keeping in mind that theoretically, the mold on that cucumber in your fridge constitutes some sort of biolab. The main released criteria about those lab that raised alarm was the presence of specimens, which does not imply any capability of creating a weaponized strain.
It's still alarming, but (at least in my opinion), still doesn't prove an effective bioweapon is buildable in a small-scale operation.
Georgetown has a good article on the general problems with our current "AI and bioweapons" dialog: https://gjia.georgetown.edu/science-technology/rethinking-th...
Without a whole lot of tacit knowledge no LLM can provide you, bioweapons are still pretty much out of range for most of the population. Doesn't mean we should ignore the problem, but a more reasoned approach would stand to benefit everybody.
Information wants to be free! :^)
(Also natives want to be paid well.)
Why should we trust them to control bioweapon development without regulations? They themselves are non-state actors.
The current Trump Admin can't even decide on the first question, let alone come up for a plan on the second.
The Trump Admin's EO was to ask nicely all of the AI companies to give them 30 days to voluntarily review each model before wide release, but they have also failed to do that for the Mythos/Fable release, only to get a call from Amazon's CEO to David Sachs to convince them to disable Fable (to all non-US citizens).
We elected the party of "minimum regulations" into all 3 branches of government and we will reap what we sewed.
And semi-related, how do you reconcile this caution with the recklessness on display in recent incidents like the Navier-Stokes drama?
Nuke requires a long supply chain and massive resources, so the worry there is maintaining control of all of the existing nuke weapons. Except for Russia racing towards Turin no itself into a failed state by staying engaged in the war with Ukraine, I don't see the nuke equation has changed much in recent years. Perhaps N Korea is a worry, but they seem to just want attention and power. Iran pretends to have nukes and says they want to extinguish Israel and the US, but I interpret that as posturing to maintain domestic control and Israel seems excellent at countering Iran's threats.
Chemical weapons have traditionally been the easiest to create (like creating chlorine gas from mixing common household cleaners). The trick there has always been volume and how to disperse it. I suspect within countries, police will have to deal more with LLMs being abused that way.
Bioweapons will be easier than in the past. LLMs will lower the barrier to entry, but bioweapons are hard to create and much of what the superpowers learned thankfully isn't in the training set for LLMs. I doubt there is much that can be inferred by having agents learn biology from first principles.
Ultimately, governments are responsible for policing these threats. Sadly we are currently both cutting government systems which work different aspects of these problems and withdrawing from the multinational orgs (UN, WHO, etc) who do lots of the investigating and watchdog work that underpin lots of the US's intelligence related to these fields.
The US has a schizophrenic regulation policy of AI where we both want to sell Nvidia chips to China (David Sachs) and we don't want to sell the top chips to China (bipartisan policy prior to Trump). We also have a terrible AI regulation policy where David Sachs disables models on a call-to-CEO-on-a-Friday-evening basis after Andy Jasse calls him with a scary story which turns out to be missing lots of relevant info (eg. The exploits was discovered in Mythos, not Fable, and other open weights models were able to find the same exploits).
There's not much we can do at a government policy level while Trump is snoozing through the rest of his term. So we are dependent on the AI companies to police themselves, but it's clear from this report that it's insufficient to prevent all serious abuse of the models.
All of which, rogue actors have easy access to and could have accomplished for a few millions dollars anytime in the last decade.
And yet, we aren’t drowning in our own blood while our organs liquify, mainly because building and releasing such a pathogen isn’t something people want to do as it’s pure and utter insanity. For those who are inclined to do so, they would do it regardless of whether they had an llm or not because they are, at the end of the day, zealots.
[0] https://universityresearchpark.org/uw-madison-scientist-allo...
You are missing the forest for the trees. The existing virologist PhDs and the GoF labs are a scarce resource. The model makes the same scarce knowledge accessible to many more malicious actors.
Do you remember that any Wuhan/GoF discussions prior to 2022/2023 were considered conspiracy theories? Maybe you should have intervened earlier!
Instead, now when it serves the hype and the ClosedAI/Misanthropic valuations discussions are suddenly allowed.
But you already write yourself that LLMs are useless for the task, like for any task apart from brute-forcing mathematician guided Lean proofs.
Even if they weren't useless, you still would need a lab, which are already monitored and destroyed like in Iran. By people in the real world.
I invite you to read the front matter and the report for yourself. Because from where I'm standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease.
Anthropic and OpenAI were founded by people who wanted to use AI to do good, and one of the causes I've heard many different founders talk about is ending disease. This report is antithetical to that.
I've attached relevant parts of the front matter below.
I invite everyone who is reading this to please tell me, how does stopping a researcher from using Claude to write a grant for a new anti-depressant stop "bioweapons?"
-
Note,"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"From a different case study.
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"and then,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute".
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
This report genuinely makes me upset, because if it is to be believed to the letter, then Anthropic seems to be actively harming medical research at a global scale. That's not OK.
I think Anthropic was founded by people who wanted to control how other people get to use AI, and define doing so as the highest good possible. Much like the good intent of german's national socialists in applying the latest evolutionary science...
Yeah, sure man.
Like Kabuki theater.
Is this what they call “collective psychosis?”
It sounds like a friend who's just learned a new word and wants to use it in every sentence
Really... what makes it illicit?
And yes, it makes the future really messy and all the nice little lines we've drawn on paper that make sense stop making sense.
Like I know Google can read any of my emails, but I also don't see them do monthly blog posts describing intimate details from each email they found in one guy's Gmail inbox who their algorithm flagged as "maybe possibly kinda sketchy: 70% confidence"
Sorry y'all.
https://www.theguardian.com/world/2013/aug/01/new-york-polic...
Imagine that during the Cold War US would concentrate all efforts to block nuclear research and basic physics classes, because it is unsafe, ahhh
My guess is the world police come collect all your GPUs and then they get turned into licensed munitions. People at universities get licensed access and the rest of get functionally retarded models.
"DeepSeek serves Claude instead of its own models and collects exchanges for model training"
Obviously. This is how they were able to score so high in benchmarks.
These companies have proven they are willing to distort the truth, or outright lie, in order to inflate their valuation / protect their position / continue the hype-machine. Nothing they say can be trusted.
The next day, the ant's nest was gone.
In hindsight, it was almost certainly the mushrooms. Thank goodness we didn't have the kind of kids who would have dared each other to drink some... that could have gone legitimately badly.
Decades later, I mentioned this to my father and he recalled that there was this ants nest that he had intended to take care of, which he remembered for that long to give a sense of how out-of-the-ordinary this was. He was surprised when it just disappeared entirely one day, and perhaps just as surprised to find out decades later why it just disappeared.
Nobody needs to report me... I'll turn myself in.
I have to admit I posted this just so I could use the word(?) "formicacide". It seems an opportunity unlikely to arise again anytime soon.
https://news.ycombinator.com/item?id=49646988
This apparently is the backup submission.
(And by "they" do you mean Anthropic? How would they have the ability to do that?)
Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease. This report is the antithesis of that mission.
From the report, presented with highlights and minimal commentary,
Note,"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"From a different case study.
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"and then,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute".
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
Reading this makes me feel upset. From where I am standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease. Because "bioweapons."
They could easily use a Chinese model but they didn't.