One that I'm willing to share (albeit from just a week ago) - I built a Python library last week that bundles MicroPython compiled to WASM to create a sandboxed code execution library: https://github.com/simonw/micropython-wasm

I just told Claude.ai (not even Claude Code - this was the standard Claude chat interface) running Fable 5:

  Clone simonw/micropython-wasm from GitHub
  and research how this could use a full
  Python as opposed to MicroPython

  uv run --with https://static.simonwillison.net/static/cors-allow/2026/cpython_wasm-0.1.0-py3-none-any.whl \
    cpython-wasm -c 'print(45 ** 56)'

(It's possible Opus or GPT-5.5 could have done this too, I've not tried the exact same sequence. The Fable vibes are good here, though.)

And that's the thing. These comparisons are all gut feelings. I'm missing objective unbiased measurements to actually have real comparisons between different models, their different generations, or even just the convention that everybody adds "you are an expert software engineer" and "don't make mistakes" to their prompts because they think it improves anything. Nobody knows if it actually does.

You can’t benchmaxx an eval that comes after your model release.

Consider also benchmaxxing makes no sense from an incentive structure: the quality of these models is directly correlated by how well you can measure true performance in the wild. If they were just stupidly benchmaxxing they would be unable to do trustworthy ablations or know how well the model will perform in their product.

Remember the famous case of asserted benchmaxxing from llama 4? The entire org was gutted and the ceo spent billions hiring better people. Every lab takes evaluations extremely seriously.

Sure you can, just do it silently and don't tell the people hitting your API that the model is different now. Unless it's open weight, we're just taking your word for it. Even better, do a VW and try to detect which benchmark is running, then change to a hyper specialized model that is trained on it.

This is...just incredibly conspiratorial and a bit silly. You can make a benchmark right now and run it on the models. They'll have a benchmaxxed model on your...previously non-existent benchmark? I mean: if models really were overfit to benchmarks, which zero lab is doing because its idiotic, against their incentive structure, and easy to detect, then why would we see a slow ascension of performance on say humanity's last exam for one benchmark example? You could trivially get those numbers to close to 100% if you wanted to.

That’s why students are evaluated by teachers with more knowledge and experience than them. It follows that any mechanical evaluation scheme is hopelessly inadequate for measuring the true capabilities of a frontier language model.

This starts to break down in college when the professors often at best only slightly ahead. (they have more knowledge and experience - but in a slightly different area and so it isn't relevant to the depth of whatever is under consideration) Grad school is about advancing the state of the art - if you don't know more than your professor you are doing it wrong.

I can't speak to the humanities, but this estimation is just not true at most universities in the sciences. (EDIT: As cycomanic emphasizes below (https://news.ycombinator.com/item?id=48477683), the part of the original comment pertaining to graduate education is more reasonable. I am speaking here only of undergraduate education.)

> This starts to break down in college when the professors often at best only slightly ahead. (they have more knowledge and experience - but in a slightly different area and so it isn't relevant to the depth of whatever is under consideration) Grad school is about advancing the state of the art - if you don't know more than your professor you are doing it wrong.

How is this remotely true. You can have verifiable tasks that you can’t do. Where does this idea come from??

That is what benchmarks and intelligence tests are, which are vulnerable to benchmaxing etc. You wont be able to do this by gut feel though, you can create a personal benchmark though.

But point was that personal judgement of intelligence requires high intelligence. Creating a benchmark doesn't require as much but is more vulnerable.

Sure you can create a personal benchmark. Who will evaluate it, you? How many tasks will it have? How will you evaluate success? Will you know which model is which or will you be blind? Which one will you do first? Ah right, benchmarking.

Also, benchmaxxing isn’t possible when the benchmark and measurements come after the model is released, right?

throw the same prompt at multiple models and see how far each one gets. change the prompt used in the benchmark every day so models can't be optimized for that one prompt. use your vibe glands all you want, but don't issue model judgements without any ability to compare apples to apples.

Ha, of all examples you had to pick this :D I think we can very well determine that qualitatively.

Adding "do not make mistakes" is silly, in my opinion. There is always a good chance it will make mistakes. You should rather be more specific about a thing rather than as broad as "do not make mistakes" is. It just does not work that way.

So the best we can do right now seems to be to combine imperfect case studies like this with imperfect benchmarks to get some unreliable impression of where we are...

https://generative-ai.review/2026/06/claude-fable-rush-test-...

As mentioned in another HN thread I've done a qualitative side-by-side measurements of Claude Fable vs Opus 4.8 vs ChatGPT 5.5.

Anyone is able to check the output for themselves and form a judgement.

Large visible improvements for Fable over Opus 4.8 and ChatGPT 5.5.

I recently did the same to show the progress from Opus 3.4/ChatGPT o3pro one calendar year ago.

That website is 95% not you, it's AI, and I feel that's causing you to way over-represent the value of it in your response here, or you're completely misunderstanding what the person you're responding to is asking. If you put all of your effort into that site, without AI, it would be infinitely more valuable and useful.

The person you responded to asked for specific things, including:

- obvjective, unbiased measurements, but all that page has is side by side visual comparison of outputs.

- their different generations, but all you included was the outputs

- details on the prompts and little things people are adding because they feel they need to, but you didn't include any of that

This is slop, it's the exact sort of self confirming fluffy AI stuff that other either inexperience or over-invested-in-AI engineers will look at briefly, skim, see quick visual validation, and nod, noting down how much better Fable must be without getting any actual data.

Sorry, it's early, and maybe this is a misplaced rant, but the person you responded to specifically asked for precise, quantitative things precisely because everything else is fluffy slop like this, and people don't even recognise they're doing it any more.

Fable just got announced and I did a rush out article because people are curious. I released the post mere hours afterwards and it takes time to create the output, slice into videos, make a wordpress article on top of taking my son to basketball training and eating dinner. I’m in London and this was all happening at 1am.

If you check the links my previous articles have all the juicy stuff you are criticising me for not having with little preparation.

How is a side by side direct comparison NOT precise?

[1] first in series from 2025: https://generative-ai.review/2025/05/vibe-coding-my-way-to-e... . This has all the background you are talking about in the Appendix

.

[2] https://generative-ai.review/2026/05/vibe-coding-my-way-to-e... . Second in series 2026 has a side by side table of what changed. This is what is possible with more than a few hours advanced warning.

I just read the extra link you provided which has some more information, thank you. Sorry, but the links confirm my points. You're not giving any quantitative analysis of your use of the different LLMs or your process. Your "sciencey appendix" is all about the domain science of pyramids, nothing to do with how or what you put into the LLMs, or any quantitative analysis of the code put out.

I'm sorry, your response has just proved the point that frustrated me: you've either lost or never had the capability to recognise a decent quantitative assessment of technical software creations.

Your entire site is obssessed and fixated on the impressive looking outputs of LLMs, rather than actual quantitative assessment of the quality of the outputs. This is the killer problem of AI: it looks like it's good, and a lot of the time, things that look good are good. It's very easy to make stuff on a computer that looks good but isn't for various reasons, and I nothing in what you've said here suggests that you fully grasp that. Sorry again to be harsh here, this is just my opinion, and we're probably going to have to agree to disagree.

In my opinion, if one cannot express themselves civilly, they should refrain from commenting.

AI is a powerful tool and very capable of - amongst other things - making something look far more valuable than it actually is, and that is a huge waste of time that costs us all. We all have a responsibility to call this out when we see it.

It looks like you've just implied I'm entitled, unhinged, uncivil and and that I shouldn't have contributed at all, whilst thinking you've elevated yourself above that behaviour by saying "in my opinion" and "one should...". I think that's an unhinged, insulting and uncivil way to express yourself.

I don't think it was "a huge waste of time" or needed your rant.

You called it slop and questioned the competence of the author, as if he made grand claims about the objectivity of his comparison.

What I see often is that people assume others are incompetent just because they used AI, when in reality they are engineers no less competent or experienced than others on this website.

I raised this in a harsh, but repeatedly apologetic way. The person then responded telling me to "get my facts straight" and doubled down with more weak, qualitative outputs of LLMs.

I don't assume the person is incompetent because they used LLMs. I use them daily. I'm a firm believer everyone is an idiot, just in a different subject.

The issue here I feel is that LLMs are increasingly leading people think that they're not an idiot in any subject at all, and when real humans question it, they double down with more AI stuff.

https://www.anthropic.com/news/claude-fable-5-mythos-5

And there’s no reason evals can’t be done on multi-turn agents in a loop (or not): it’s pretty much what all these benchmarks do.

It would be very useful for companies to isolate interesting programming challenges in their past and publish evals on them (without revealing the actual codebase). In theory companies adopting these models should already be doing this to evaluate cost/benefit for each model, so it would be a matter of publishing them on a regular basis.

"Don't make mistakes" does seem dumb. It's not guidance.

https://simonwillison.net/about/#disclosures

"I have not accepted payments from LLM vendors, but I am frequently invited to preview new LLM products and features from organizations that include OpenAI, Anthropic, Gemini and Mistral, often under NDA or subject to an embargo. This often also includes free API credits and invitations to events."

But I'm totally unbiased on my gut-feeling posts, trust me bro.

-- AI influencers.

[1]https://en.wikipedia.org/wiki/Simon_Willison

Fable is doing - so far - a great job. I just had one big question around how part of it should work. I had a design sketch, but with some big unknowns. I asked fable to figure it out via reasoning and prototyping, and it did - it even, under its own initiative, wrote a fuzzer for its prototype which explored and verified that its reasoning was correct. It absolutely nailed it. And it found, and fixed, a couple bugs that I'd missed.

I'm sure its weaknesses will become apparent in time. But, wow this thing is a beast. Its the first time I'm reading the work of an LLM without spotting obvious weaknesses in its reasoning and code. I'm really impressed.

I work on the live collab at my company, and using AI while coding has into recently sort of “clicked” for me. We use an (I’m pretty sure) unheard of algorithm for collaborative editing, and I’ve had a long term goal of turning it into an implementation of EG Walker, but our document model is very complex and most out of the box CRDTs don’t quite fit. Maybe Fable will be what gets me over the hump.

https://blog.helsing.ai/posts/dson-a-delta-state-crdt-for-re...

https://www.youtube.com/watch?v=4QkLD7JhD_I&pp=ygUJZHNvbiBjc...

For such a data structure, "nailing it" means a formal proof of correctness. Fuzzing, as useful as it is, is merely throwing dirt at the wall and seeing if anything sticks.

I’ve read plenty of papers with “formal proofs of correctness” that turned out to have huge flaws. Machine verifiable proofs I trust. But I’ve personally found more bugs with fuzzing than I have via proofs.

I have found this quickly becomes false. I have learned I cannot review llm generated code as if it is written by a trusted senior developer (where I often just do a quick look, see nothing obvious and hit approve). Once you start reading the code in depth with the goal of understanding you quickly see the places where flaws are likely. Sure I start with no clue where to look, but it doesn't take long to see things.

I saw scanning the comments and saw you mentioned CRDT. Just wanted to mention that I implemented a CRDT-flavoured sync engine for the product I'm working on a while ago, I think it was with Opus 4.6 if I'm not mistaken (or earlier) so it's not something new to Fable 5, just fyi.

So far at least - and its been less than a day - Fable seems better at this.

I think I also do my CRDTs differently from others. I've grown to like the pure-oplog approach after making eg-walker. LLMs are much worse at this!

Damn you must be good, I've been feeling this for around 2 years now

"But it doesn't do well when writing my undertrained language" - yeah, fine. Yet. Reasonable code in that is probably one RAG + verification scaffold deployment around Mythos or maybe mythos+1. Just like it was for you learning it, because you knew how to _program_.

AI is just another tool, learn to use it.

Like it did everything:

- this is not a Linux system (true, it was macOS) - it is not an available command - the binary is corrupted - node/js is more precise - V8 JavaScript is faster than bash (true technically??? But not in this context lol) - JavaScript is more versatile

I forgot what else we went through but there were a few more things. I indulged it because it was incredulous and funny. The prompts from my side were all questions, never instructions. I assume an instruction would've helped here, but also I don't think Opus ever did this (but on the other hand Opus wrote python scripts to format/indent, instead of just running cargo fmt, so I guess potato potato)

> Fable 5's safety measures flagged this message for cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Switched to Opus 4.8. Send feedback with /feedback or learn more

I'm working on an internal tool that does new business prospecting data collection, scoring, etc. This is ridiculous.

https://x.com/i/status/2064449457869984035

Assuming the model is being “truthful”, CC is just being stupid in its detection mechanism.

I’m having a really hard time believing some weak reason for a 30 day retention policy.

e: I quit the session and went back in. Set it to Fable and told it to continue the last session. It's moving along as if none of that had happened.

How weird.

https://www.wired.com/story/openai-anthropic-letter-ai-biolo...

Or Fable’s arch is different enough the allocated clusters of compute targeting a date, and here we are, ready or not.

Or…

Question is if there will be any competition in this area...

I see a lot of people saying they are happy with weaker models, but I am the opposite, I need more strength, more intelligence!

I am quite happy that opus 4.8 can do some medium intelligence problems. And maybe Fable 5 can do some more more of those! I have a lot of problems to solve!

At work I had to switch to using GPT 5.4 Mini and Qwen 3.6 27B.

The results were near useless.

The error rate is through the roof, it's constantly incorrect in its conclusions even when investigating very simple issues.

Further the models are too unreliable to even move 20 line snippets around without inadvertently modifying them. Ask them to correct it and they still get it wrong.

Maybe the larger Chinese models are better, but the Mini stuff is next to useless to me.

I am just testing it on stuff I know intimately myself. I would probably not understand a proof of Collatz if it was dansing in front of me!

The curse of the 'use case' comes in here too. When people think that everything should have a use case, that's a lot of training data suggesting to a model that things should only be used for what someone has already thought of.

A couple of times I have had to manually code proof of concept pieces so that the model breaks out of that "unpossible" mode and actually helps me.

I can't remember if it was chatGPT or Claude, but when I showed it how to get a MessagePort in its JavaScript executor through to the artifact/canvas, it quickly went from "That can't be done" to positively enthusiastic about the possibilities. I suspect those shenanigans will be well off the table for Fable though.

Sorry to belabor this but it's basically pointless saying you have nuts it can't crack without showing us the nuts.

I gave a high level description of the problems in a sibling thread. They are the kind of small problems which I suppose every researcher has lying around, waiting for them to think about some day. But not the big problem everyone is waiting for to be solved.

My comment was not meant to be a tease – sorry! I assumed there would be other people in a similar situation, who might relate.

(Joking aside, see sibling threads.)

Recently (last couple of months?) these models are becoming useful tools for mathematicians, because they can solve easier problems more quickly, meaning that one can tackle bigger challenges (but maybe not RH et al) piece by piece.

But, there are still definite limits, where one could expect an expert human to solve things, given time, but models do not. Thus, more intelligence would be nice!

These are not Fields medal type problems, nor know difficult/open conjectures. Just small stuff I have collected in my todo list over the years.

A year ago my judgement was that I had wasted my time on trying to work with the models and doing things myself would have been more productive as I would have gained intuition from the failures. Now it definitely seems to have figured out stuff that would have taken me more time than I have to spare on this problem...

Being a theory builder more than a problem solver I am excited for the future.

Also excited for fully formalised mathematics to hit main stream!

I’ve done the same thing with opus multiple times with no issue. According to ccusage I racked up just shy of $100 of tokens using Fable.

It spun up subagents or workflows or whatever so obviously that contributed but “double opus” was not my experience. I’ve done the exact same prompt with opus on the highest setting and only once before (not even while using this prompt) hit my limits.

My prompt? I’m not a prompt wizard or anything but it was literally:

> Please review the uncommitted code in this repo for bugs/issues/code smells.

I use variations on that all the time with opus and never had issues. I figured it was a good one to kick the tires with Fable. Little did I know it would mean no more Claude Code for the next 4.5hrs (unless I wanted to pay) after this being the first time I had used CC that day (yesterday).

All in all, a pretty crappy first experience.

  uvx agentsview usage daily

And run the command again. I get $126.89 for yesterday.

    DATE        INPUT    OUTPUT   CACHE_CR  CACHE_RD   COST     MODELS
    ----        -----    ------   --------  --------   ----     ------
    2026-06-09  142015   85315    321224    6880110    $10.96   claude-fable-5, gpt-5.5, claude-haiku-4-5-20251001

I think the $10.96 is coming from gpt-5.5 since I switched to it once I exhausted all my usage on CC. CCusage reports completely different numbers so I don't know which one of those is right.

Thanks for trying, for yesterday ccusage says "$92.02" for claude, which I assumed was the Fable usage.

  uvx agentsview serve

Unfortunately it's not telling the whole story. The last message from the _only_ Fable session it monitored was:

> The data layer looks clean — <REDACTED>. Now waiting on the 11-angle workflow — verification and the gap sweep run after the finders; I'll compile the full ranked findings list when it completes.

And my memory jives with that, I could see in the footer that it had spun up 11 agents (though agentsview says it used 0 subagents, don't know if it was "actually" workflows that it spun up?). It's like it didn't record the sub-sessions/sub-agents info?

I'm still shocked that my prompt (which I now can see thanks to this tool) of:

> Please review all the uncommitted work in this repo and identify any issues.

was able to burn so much, so quickly, and, most frustratingly, without actually doing anything useful because killing it was my only option lest it spend even more of "extra usage".

Overview of usage: https://cs.joshstrange.com/RjGzWVXy

Stats for that 1 session: https://cs.joshstrange.com/Fj5qv1wl

https://cs.joshstrange.com/z9x6SPcC

I've been watching my usage quota bars drop as I use the model, so I don't think I have a weird quota issue going on here.

Soon the times of AI for $20/$200 a month will be long gone.

Forcing developers to pay for models that were build on code they scraped scott-free

A tax to do their job that developers are jumping at the chance to pay

Everybody's finally realising that node dependencies are a threat, but letting these AI companies gatekeep the industry is a bandwagon people are scrambling towards

Yes this makes me sad behound explanation. Specially when I see open source developers happily using these tools. These companies stole your, free, hard work and charge you a subscription!! Not to speak about them torrenting books and (most likely) training on private repos.

This and devs paying a subscription to use a tool that is marketed as trying to replace them.

I had 150$ monthly budget thatbI used for various open source projects and I've cut that entirelly.

In case you weren't aware, Anthropic, OpenAI and GitHub Copilot all have programs that provide access to open source maintainers for free:

GitHub: https://docs.github.com/en/copilot/how-tos/copilot-on-github...

Anthropic: https://claude.com/contact-sales/claude-for-oss

OpenAI: https://developers.openai.com/community/codex-for-oss

Then you say you had money that you used to donate(?) to OS and have cut that because of the frustration?

Open source just means sharing the source code for people to learn off or have the ability to customize on their own. I don't think there is any need to be frustrated about that (now if it was copyright/private of course).

Yes people, not corporations. The point is there a licenses to be respected that weren't.

We could fix that, but it requires a political will to change the law.

> To summarize the analysis that now follows, the use of the books at issue to train Claude and its precursors was exceedingly transformative and was a fair use under Section 107 of the Copyright Act. And, the digitization of the books purchased in print form by Anthropic was also a fair use but not for the same reason as applies to the training copies. Instead, it was a fair use because all Anthropic did was replace the print copies it had purchased for its central library with more convenient space-saving and searchable digital copies for its central library — without adding new copies, creating new works, or redistributing existing copies.

That's also caused by some very smart (even brilliant) developers (you can see many of them in this very thread) choosing to be oblivious about all this and bury us all under, hoping that they'll be among the last ones to go. Writing this down I realise that they maybe aren't all that smart.

It would not surprise me one bit to see anywhere from $80k-$100k/seat pricing.

Not everyone needs a Ferrari to go for a weekly shopping.

Maybe? If you talk to executives, the impression that I am getting is that they tend to be somewhat misinformed at best, which, yes, is bound to result in some really bad decisions down the road. But, and it is not a small but, the ones I did talk to ( and, amusingly, those are the ones with strong opinions ) don't seem to have a lot, um, practical exposure to this tech beyond what they heard at the watercooler. Honestly, it is kinda infuriating. And all this before we get to how companies want to say they use AI, but also keep cost down.

You and your work are not that special, you are not participating in car races, and you don't need a Ferrari.

So AI is only interesting to you / your org / humans if it can do things that you can not achieve. But if it still does errors, how could we ever know that super-invention by AI is not wrong?

If we can not rely on the correctness of the result, it is not usable at all. AI must create reliable and correct results always. That was a very fundamental requirement for computing. This problem has not been solved.

AI is interesting as long as it can save time and/or money in getting an acceptable result. Anything that runs on a computer and can do "things that humans can do" will automatically end up doing things that humans won't do, simply by virtue of the fact that it runs on a machine that doesn't require sleep, doesn't get bored or demotivated, etc.

Verifying code (to a level where a responsible person is willing to take ownership for it) isn't trivial, sure; but writing the code by hand requires the same level of care, and the fact that the same person wrote it doesn't actually allow for shortcuts (if we're being properly responsible).

What if an LLM overall starts to make less mistakes than a medium developer, costs less than its salary and is 100 x faster? For sure, the companies that will leverage these with just a few senior devs doing prompting, testing and requirements analysis, will outcompete other organizations.

AI agents do that, perhaps not always, but still do. Now the question: would I trust AI without verifying its output?

Do you verify every line of code written by your fellow developers? I doubt it, which is strange because they make errors don't they?

What matters is the error rate. Past some threshold and they're better than senior devs who you don't supervise closely.

No idea what's going on here but agent tested a bunch of stuff. Then I asked to build a wheel so I can run the command you noted above and it appears to pass

For those who are curious...

https://github.com/bamggm/micropython-wasm/commit/5ddebae592...

https://github.com/bamggm/micropython-wasm/commit/8b362fba1f...

To be clear, the jump from Opus to Fable was like the jump from pre o3 -> o3 for me. Very sharp improvement, not incremental. But that could be explained by dummy long thinking times.

It one shot a task that Opus burned hundreds of dollars on to get nowhere. Very tricky semantic refactor, got it right. Granted, again, the semantics Opus and I fleshed out 3 months prior, but Opus couldn't execute on the vision. Fable could.

Then I discussed some philosophy and it was actually both pleasant (GPT constantly "corrected" you for the sake of correction without clarification, also still often just wrong; it's like it refused to think critically about philosphy) and accurate, and actually helped resolve some deep but subtle misconceptions I had around representationalism. When talking with GPT I felt like I was talking with someone who either was sycophantic or "anything that is not absolute truth is relativism" - Fable actually discussed.

Both is exciting and kind of makes me depressed. I can definitely see why people are getting hyped about AGI again. All the models were extremely strong technically but I felt like couldn't match the developer's tacit state - Fable definitely did, and that's a basic quailty to be considered "usefully intelligent" IMO, at least to me.

Shame that it's going away in 2 weeks and probably going to be nerfed if/when it's re-released.

[0]: https://github.com/eryx-org/eryx

Which has a full build of python to WASM with a bunch of static libs built in already.

I will say I built this pre fable and actually the first build of the interpreter to WASM opus pretty much nailed, cpython has secondary support for WASM as a target since like 3.9 or something and it just pulled from that.

I’ve been meaning to write up a blog post about this sometime, building this has been pretty interesting, including using opus to run a full auto research like loop for days to hyper optimize it’s performance.

I’m hoping to use fable to power some even crazier WASM adventures tho.

It feels like you can give it a big chunky problem and leave it alone and it gets it done, with less questions and fewer design decisions that I wouldn't have made.

In reviewing its code I'm finding less to complain about than Opus. But it's all vibes, if you want a more scientific comparison you'll have to look elsewhere.

https://generative-ai.review/2026/06/claude-fable-rush-test-...

I get them to make a 3D explainer animation. You can clearly see Fable is much improved on both Opus 4.8 and ChatGPT 5.5.

Better Textures . A nifty camera follow . Humans rendered better . ... see for yourselves

> But it's all vibes, if you want a more scientific comparison you'll have to look elsewhere.

Fable just did it, clean code, one timeout with a hanging bash script, fixed a couple very old very structural bugs in the codebase

I am not sure it's perfect, and it will need further validation

This morning I looked at code samples & checked if all unit/integration and e2e pass & perfomance tests pass

I also generated a postgres schema diagram.

Aka I did probably 2 hours of work, rest was not me

The opus try was last month

It made sense for people doing proper and fair AI breakdowns waiting on an embargo, but now it's just slop I don't trust anymore.

> I have not accepted payments from LLM vendors, but I am frequently invited to preview new LLM products and features from organizations that include OpenAI, Anthropic, Gemini and Mistral, often under NDA or subject to an embargo. This often also includes free API credits and invitations to events.

[1] https://simonwillison.net/about/

Compared to what?

Update: looks like I've spent $82.92 in Fable 5 API priced tokens so far today (still all included in my subscription.)

Here's a TIL on how I'm calculating spending using AgentsView: https://til.simonwillison.net/llms/agentsview-custom-model-p...

* From today through June 22, Fable 5 is included on Pro, Max, Team, and seat-based Enterprise plans at no extra cost.

* On June 23, we’ll remove Fable 5 from those plans. Using it after that will require usage credits. If capacity allows, we’ll extend the included window.

* After this point—when sufficient capacity allows us to do so—we aim to restore Fable 5 as a standard part of subscription plans. We intend to do this as quickly as we can.

It's been discussed at length (on this site, on other sites, on like every blog ever, etc) that, eventually, those subsidies will end, much as the $5-10 Ubers/Lyfts I used to take from the far north end of Chicago into the Loop in 2016 would eventually end once those companies had a footing and didn't need to hook folks.

So - yeah, I mean, a v5 model launching in a year where Anthropic has a rather deeply established market and in a year where AI costs are rising from nearly all providers (sometimes for multiple reasons) seems like exactly the thing I'd expect them to pull the subsidy plug on after a launch teaser.

(Even the open-weight models sometimes do this: for example, OpenCode Zen/Go has a rotating door of free models at any given time that eventually leave the free tier and move into the paid tier once the launch day hype/marketing dies down)

Also, a fun website: https://isaiprofitable.com/ (thr numbers are probably made up)

(You may not realize it but simonw is one of the cofounders of Django, Python's web framework. If they find a Python problem difficult, it probably is.)

Web development is not a domain I would consider noteworthy of making a framework given how much development there has been in that area.

It's frustrating that superfluous tokens are burning up our quotas:

key insight, crucially this, real engineering deltas, net assessment, definitive picture, acid tests, real limits, sharp boundary, proper patch, real root cause, big progress, actually wrong, path finagling, the catch, root cause pinned, everything passes cleanly.

Though that's also what makes humans so good at solving problems as well, it turns out.

Also, slight tangent: but I do find the "clanker" insult kind of funny. I feel like it counter-intuitively makes the models sound cooler than they are, if anything. I love clankin' shit.

Next time you get a new and a fresh and an inspiring idea, and you spend hours solving a unique problem nobody has ever done before. You can take comfort in the fact that a few months later some lame and uninspiring developer can write the same problem in a prompt and get the plagiarism machine to steal your work, just in a more lame and uninspiring way.

That may very well be true now. And in fact, this was true of more rudimentary calculations early on in computing history, where humans were definitely more efficient, particularly for more abstract mathematics. But Moore's Law comes at you fast. Even without more efficient compute, it's rather wild how much more efficient models are becoming these days just from algorithmic and training improvements.

So, maybe for now, certainly. Are you confident that will be the case in 5-10 years? And is that really your barometer for success?

>And when a human learns these things they usually remember how to, and are able to extrapolate that knowledge into new and fresh problem spaces.

That is certainly a limitation for now, but plenty of academic research is being done on how to address that in a more individualized way. That said, the models also have the advantage of synthesizing learnings from user interactivity back into a future release and essentially applying that globally, which is pretty neat.

There's also some cool techniques to sort of bridge the gap today, like compound engineering.

>Next time you get a new and a fresh and an inspiring idea, and you spend hours solving a unique problem nobody has ever done before. You can take comfort in the fact that a few months later some lame and uninspiring developer can write the same problem in a prompt and get the plagiarism machine to steal your work, just in a more lame and uninspiring way.

But that's the thing: it's becoming pretty clear that the "plagiarism machine" can probably take that same problem in a prompt, having never been trained on my code, and still solve it.

In that case...maybe it doesn't feel great to have someone copy my idea. But that is certainly not plagiarism in the way you mean it. And when you put ideas out into the world, you can't be certain that someone else won't copy and remix it into something new. That's kind of how the world works already, but we're just seeing the barrier to entry decline.

Yes, I am. I am very confident that general purpose digital computers will never be more efficient then human minds in generating moderately complex code.

Why am I so confident... Well, it has been over 10 years since AlphaGo beat top go player Lee Sedol. AlphaGo was able to beat the a world class go player by doing several thousands orders of magnitude more computations then Lee Sedol, and it did so by spending several orders of magnitude more energy then the top human go player. Today, over 10 years later, the top go machines are able to beat world class go players much easier, but still do so using the exact same strategy of outcomputing the humans with thousands of orders of magnitude more computations, and spending orders of magnitudes more energy.

Things did not change in the past 10 years, I see no reason why it should change 10 years from now.

Has it not? Why do you say that?

Also, do we still require a Deep Blue sized supercomputer for chess? :)

But regardless, compute will get to a point where human level intelligence close to as efficient as we are. You could argue it already is today, when you factor in the resources that the average person in the west already uses in terms of their overall impact on the planet.

I can just as well describe the future evolution of the internal combustion engine and claim it will get more and more efficient and eventually we will be able to burn oil so efficiently that our personal vehicles can fly through the atmosphere at twice the speed of sound.

There is limitations to digital computers just as there are limitations to internal combustion engines. Our brains are not digital computers. When we use our brains we don’t just do a bunch of linear algebra.

This is a silly comparison. There is a certain quantity of energy stored in oil, so we know what peak efficiency looks like. We don't actually know what amount of energy is required to solve certain problems. We quite literally have models with quite a bit of capability that can run locally on a phone today, right alongside Stockfish, for example.

And this is to say nothing of work happening now on new hardware approaches, such as Normal Computing's work on thermodynamic matrix math: https://www.normalcomputing.com/blog/a-first-demonstration-o...

That said, this feels like a strange tangent: I'm not sure it's that important that the models be as energy efficient as a human brain. We don't avoid cars because they're less energy efficient than our legs. ;)

This matters because unlike cars LLMs are only doing stuff we can already do using our brains, just several orders of magnitudes less efficiently. Cars can at least take us distances we would never be able to using our muscles. In comparison, if I need to compile CPython into a WASM binary I can simply download a library that does it, or copy paste code in a few seconds, for a million billionth of the energy it takes an LLM to do the same. Except when I download the library or copy-paste the code I (hopefully) attribute the original author and give them credit for their work.

OK then - do it, faster.

> You can take comfort in the fact that a few months later some[...] developer can [solve] the same problem [using your work]

Isn't that what collaboration and sharing software is supposed to be all about?

On the other hand: "Stop trying to make 'clanker' happen! It's not going to happen!"

"AI slop" caught on but "clanker" did not.

It caught on, sure, but not exactly in the way I expected. The wild popularity of "slop" as a term for AI eventually gave way to the genericization of the word "slop" to mean "content of low quality, regardless of source", and is seemingly being used as just a derogatory term for anything that people dislike (particularly by folks in left leaning communities). For example, I've seen people refer to (clearly human written) commentary from some political commentators as "slop".

You comment kind of reinforces the idea by the fact that you have to now say "AI slop" specifically to disambiguate it. It's kind of a fascinating little turn.

We're a society built by thought and good-will engagement. We won't get out of our "rules for thee" with less thought and less good-will engagement.

So Fable would cost me 20k/mo at Enterprise rates. That’s around the average cost of a loaded SWE in the USA. “But I’m >2x more productive” doesn’t justify doubling the opex of the Software/IT department for most companies when revenue isn’t even up 10%.

I switched to DeepSeek v4 Pro with OpenCode and am on track for a few hundred dollars of spend this month.

Rewriting your stack from Ruby to Go in 2 days where it would’ve taken 6 months is impressive and fun. But that isn’t upping revenue.

Iterating on net new business features and ideas that are niche that the LLM isn’t trained for are much harder. Is 20x the token cost worth it there?

So this pricing is just completely outside of our economics and nobody I know would pay that, no company will justify spending $20k/month when they can hire 10 more developers instead.

It is very interesting unfolding of events. Can't wrap my head around it completely.

* Average software dev salary in Q12026: 4945€ / month [1]

* Total cost for the employer: 6616.41€ [2]

For $20k/month, you'd get 2 x full time mid-level developers + 1x junior dev or QA.

So the calculation becomes: which option can produce better results for your specific use-case, "you + Fable" or "you + 2x mid-level developers + 1x QA". (and from personal experience, mid-level in Estonia = senior dev in the US, in terms of skillset and experience.. but YMMV)

(Of course that's simplified. Your full time devs need _some_ level of AI subscription as well + hardware so add a couple of hundred to their salary per month etc so you might only be able to afford 2x mid level devs, instead of 2.5)

[1]: https://palgad.stat.ee/en

[2]: https://www.palgakalkulaator.ee/en

(Our team spend on AI devtools comes out to around $1500/person/mo)

This is a good start, but the calculation doesn't include office space and overhead (for every 100 developers there is maybe 5-10 support staff to cover the additional legal / administrative, and don't forget the extra cost in supervisor time to manage them)

one big enough to license the model and self host on existing infra.

Hitting the first calculator I found gave me 50 kSEK costs 69 kSEK. So far from double nowadays.

I understand pension contributions, but what are the other "hidden" costs that could equal the net salary?

The employer pays £6k for National Insurance (atop the employee's NI contributions). Pension: 2-3k. Apprenticeship levy is £300. 3yr-amortised recruitment fee is £4000. Hardware costs: £1000. Office space £5000. Software/tools: £2500. Benefits: £1500. Training: £1000. Other admin overheads £500.

You pay that person for ~250 working-days, but they only attend for ~220, due to annual leave and sick pay, so you get around £62k worth of attendance out of that person in exchange for £70k, of which the employee sees £35k.

So your "£50k" salary actually costs your employer £56,750, and that's before all the other expenses mentioned elsewhere in this thread such as hardware, office rent etc.

This is not visible on your payslip, i.e. if you earn 5k€ brutto, the employer has to pay these shares on top of that.

Our top user is at 10k a month, but the next highest is $2,000.

I would say the average is around $1,000-$1,500 for a developer.

We have completely unrestricted access to Claude, Codex, and Cursor.

Funny enough, the guy spending 10k is not even a dev by trade but an SME in what we work on that just vibe codes apps and somehow has not been cut off yet lol.

I have a single thread of GPT 5.5 medium running basically all work hours and I am around $1,500 a month in spend on Enterprise pricing.

I’ve heard of a few cases of devs racking up bills fast, but it has typically been due to inefficient context usage. Like they just have one super long session with Opus 1M and are getting killed with input token costs and cache misses.

With careful context management and some thought into good approaches to problems, I have personally only rarely even hit $1k in regular use.

I'm guessing he's producing pretty valuable work. We have a few SMEs that vibe code tons of stuff with Claude. The only thing they really need tech for anymore is deployment and helping get their wheels unstuck on occasion.

Multiply this times many, many companies, and you can see how providing AI could theoretically be a good business to be in. Margins may be tight, though.

Also -- I'm convinced someone will figure out more use cases beyond software programming, which will result in many more companies spending $1k+ per employee per month.

It remains to be seen how much of this is a bubble.

All of the above, of course, depends upon Fable consistently being a 2x-3x SWE at minimum.

It imitates applying knowledge. The imitation may be uncanny, but assigning LLMs intentionality and ToM is a category error.

By analogy, consider that many have referred to classical, deterministic computing as some kind of "thinking" for the last half century+. Does this stop being kosher when the computer has an uncanny propensity for human language? Perhaps, but the computer is still clearly chewing through problems that would have required a lot of human thinking (e.g., arithmetic) in ages past.

I haven't seen any genuine proposals for words to replace the human mind analogues, let alone proposals that the anglosphere would plausibly adopt en masse.

This is not correct. LLMs interpolate in a high dimensional space, so you're actually composing the best matches in a compressed corpus to find novel points/paths in that space. That is problem solving.

Depends entirely on the domain. If you're selling entreprise software, this kind of stuff barely matters for sales.

It can reduce operational costs which is good but there's a limit to how much that's worth.

This means if the deepseek / under 1k alternative is at least x1.2 improvement, fable needs to be x24, which I think is very2 unreasonable. It is possible for it to worth if it can x2 a $20k SWE, though I doubt it can do that.

LlMs are incredible don’t get me wrong, but they are good on tiny contexts (writing a script). Not on software engineering (adding features to Chrome).

Claude keeps telling me this when I argue with it. LMAO.

No it doesn’t and will not be. Companies have not realised the cost yet, wait till the end of the financial year and you’ll see a different direction.

DeepSeek v4 is pretty decent, and probably on par with sonnet. I see a future of hybrid models where opus or fable might be used only for complicated features or bugs, but general day to day would be DeepSeek or whatever good models that will be released later.

That's enough to buy a house in my country...

So what keeps your management from just buying everyone individual flat-rate Max subscriptions, or at least buying them for the users responsible for the sky-high token invoices?

I see a lot of comments like this but I don't understand why some people willingly pay so much more than others for the exact same service. What are you getting that I don't get as a $100/mo Max subscriber?

I was about to say that. Deepseek is just magnitudes cheaper and absolutely good enough for most things. Anthropic and co just try to milk the cow while its possible. If they cant compete with Deepseek pricing I do not see a bright future for them.

With that said, I still had the Pro plan on Claude, I didn't expect much, but it blew up my 5h allowance on Fable with one simple single prompt, and it didn't even complete lmao

Companies have to pay monthly for the harness app (codex, claude code) and the tokens are priced separately based on standard API pricing.

So Fable is just not usable for $20 plan and barely usable for $100 plan.

https://youtu.be/ngtp3v1_nCI

- They learn the domain of your product, which means long term ownership and knowledge establishes itself. If you've only ever shipped SaaS slop, you might not know, but lots of companies are solving real world problems that have no better solution. Owning and understanding the code and the domain is key.

- They will learn from their mistakes (no LLM does this).

- Human skill is a REAL moat. Once you build a team that fully understands and is skilled in the domain you work in, these people are going to be the thing that sets you apart. If some of them are particularly social or charming, let them sit in with you for meetings and watch them provide loads of value, for no added cost.

- If Claude or OpenAI is down, they will continue thinking. In fact, they will continue thinking even when off the clock! This is a neat little hack called "consciousness" where you get a lot of work for free!

- You can hire people who punch above their weight; not everyone you hire needs to be a 500k/year staff software prime engineer of doom, you can just spend some time and effort to hire good juniors/competent mediors who will think for themselves (gasp!) and get work done.

- You still get ALL THE BENEFITS OF AI!!!! They can use AI just like you can, or better!

- You get people who you can brainstorm with, which is distinctly different from LLMs because your employees are less likely to want to suck you dry in every sentence just to make sure you spend more tokens. Employees don't care if you love them, they care about the quality of their work if you manage them correctly and reward that.

- They are quite loyal if you treat them right; spend a little more on their well-being, and they will stick around, come in to work every day and deliver cool things with you.

- Humans can only manage, review and give tasks to so many agents. If you add more humans, you can handle more agents.

An expensive LLM and a lot of extra tooling gets you some of this, yes, but not all of it. With humans you can still do the expensive LLM and extra tooling if you end up making enough money anyway.

- AI isn't bound by need for rest, vacations, sick days, or labor laws

- AI doesn't bounce from company to company, taking your business knowledge with it (actually this isn't technically true based on the practices of AI companies, but that's not a technical requirement)

- AI doesn't join a union and stop work in demand for higher pay or workers rights

This is what CEOS and capitalists are thinking. For capital, the best outcome is to not have any labor at all. And if you can do that when your competitors can't, then you have a huge market advantage. (Slop notwithstanding)

I'm not saying this is a "good thing" but this is what drives the market. Less labor revenue in the long term and money printing machines.

> Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT). These interventions will not affect the vast majority of coding work. We estimate they will impact ~0.03% of traffic, concentrated in fewer than 0.1% of organizations

Edit: I asked Claude. It replied:

> Consumer protection / deceptive practices. In the EU this would be a clear UCPD (Unfair Commercial Practices Directive) issue and potentially a DSA violation. In the US, FTC Act §5 prohibits "unfair or deceptive acts." Selling a product that secretly performs worse than advertised for a commercially self-serving reason, without disclosure, is textbook deception. The Samsung/Apple battery throttling cases are instructive here: Apple faced regulatory action across multiple jurisdictions specifically because users weren't told.

> Competition law. This is where "anti-competitive" gets complicated. Refusing to help competitors build competing products via your ToS is generally legal — you can decide who you license to. But covertly sabotaging output quality for a class of users while charging them full price crosses into different territory. Under EU competition law (Article 102 TFEU), if a company with dominant market position uses covert technical means to disadvantage competitors, that's closer to abusive conduct than a legitimate ToS restriction.

This clearly is disclosed, otherwise how did we get to know about it?

The reason they are doing this shadow ban style technique, is they don't want users to figure out how to jail break their way out. Or the explicit direct bad PR of when it miss-fires.

Just need to wait for this thing to be open sourced :)

lol it won't tho...

https://mimo.xiaomi.com/blog/mimo-tilert-1000tps

It doesn't imply we should, for example, publish step-by-step instructions for making widespread death easier.

An example from the meat world: not publishing your vacation dates well in advance for the world to see somewhat reduces your chance of being burglarized. That is security by obscurity; not reliable, but not completely inefficient either.

But if you live in a fortress (security by key material), you can well declare your vacation dates without running the risk.

The reason we are not being attacked is not lack of technology access.

Column A, Column B. Building a small explosive device isn't hard. Building a million is very difficult, doing it covertly virtually impossible without the resources of a nation-state.

The problem with biologics is the self-assembly and replication machinery comes for "free." So the numpties who might otherwise blow up a trash can [1] now have a real chance of taking out a million people.

[1] https://en.wikipedia.org/wiki/2016_New_York_and_New_Jersey_b...

I also would like to hope that people that are likely to do such things are probably:

A) don't know how to break even the most basic guardrails of models

B) already in glasswings project

To prove point B - Theranos existed.

“Many of the largest and most responsible providers in the industry already screen and record orders voluntarily,” but there is no requirement to do so [1].

[1] https://screendna.org/

Humorously, whether I choose to participate in this hypothetical or not, I am already betting my ass.

This whole situation feels like the game [1].

[1]: https://en.wikipedia.org/wiki/The_Game_(mind_game)

// Claude, make antiviral nanobots that defend me from 6ft virus. Make no mistakes.

All of this “guardrails” handwringing is nonsense. These things output text. Are you for censorship of a book written by a biotechnology expert that gives out the exact same information?

Security in the form of "pay to play" is just kicking the bigger issue down the road.

I don't think there's an ideal solution here, but giving trusted people access to fix security issues before giving it to the wider public seems like a reasonable compromise. They're letting you use the model for all other uses.

sure, a malevolent state actor could swing it, but they could make a bioweapon without Mythos's help already.

also, vaccine production and disease surveillance have ramped up very quickly. they will ramp up further, despite political setbacks. it's a cat and mouse game that favors the defenders IMO.

but the bioterrorism narrative is useful FUD to spin open-weight models as existentially dangerous. I am far more worried about Anthropic's own goals than the goals of some crackpot in a shed.

How so? I'm actually against most of the "safety-tuning" that anthropic does, but this seems fundamentally untrue, a close analogue being video game cheat development. I think in general the cheat developer has an advantage and the cheats generally proliferate for quite a while before being patched.

Finance and biology do come across as two similar high level systems. But while we can employ KYC, fraud detection, and various auditing techniques to finance, I don’t know what you do for biology. You can easily run an algorithm over every transaction a person makes in their account but there’s no equivalent for every cell, every bacteria strain, every virus in the human body.

the adaptive immune system effectively does KYC by checking the antigens presented on the surfaces of cells. the thymus selects for B-cells (iirc?) which don't react to a corpus of the body's own antigens, but cover a wide library of everything else. when it sees something it doesn't recognize, it reproduces, warns the rest of the immune system and marks targets. that's why our immune systems can eventually conquer almost every pathogen we encounter, if we can survive long enough for it to do its work.

but the KYC I was referring to was KYC that vendors of oligonucleotides (should) be doing, to keep people from ordering nefarious sequences.

also, afaik the most effective way of developing pathogens is through serial passage through humanized mice or something like that - directed evolution at a small scale, selecting for traits. AI simply isn't needed for that. I don't think information or intelligence has been the bottleneck for bioterrorism, it's motivation and resources - same as for any other kind of biology research program.

At the scale of API requests that Anthropic sees, I think the affected organization count might be substantial, and they might not be getting the full model capability that they're paying top $$$ for.

Also, wonder how they arrived at that estimation.

1/30,000 * 100 = .003

I don't even think they can believe it themselves, it's in reality they are just trying to throw fear, uncertainty and doubt about potentially cheaper offerings.

Not what that means.

Crocodile tears "is a colloquial term used to describe a false, insincere display of emotion" [1]. Defending yourself against an attack vector you just exploited is between savvy and hypocritical.

[1] https://en.wikipedia.org/wiki/Crocodile_tears

The fun part is that you will never know if your neural net classification project is getting silently sabotaged because their classifier doesn't work!

With this in mind, I don't want model to be proactively instructed and encouraged to sabotage without telling me.

Anthropic is doing a better job with their model menu, most people I talk to know immediately that Opus > Sonnet > Haiku but cant tell you what the rank order of open ai models are, when to use them, etc.

Cool, good to know I can trust Anthropic.

I built it because I wanted cursor on my phone because I have two small kids and don’t want to be chained to my desk. And it’s awesome. It’s a full ide with agent chat, terminal and file system running in a remote Linux container. I can review diffs, fully manage git and preview/serve apps. And no one can ever take it away from me :)

I am watching the way things are progressing with the ai api vendors and it feels really clear that depending on them will soon be dangerous. So I an furiously building as much of my own infrastructure to capture some autonomy with these capabilities

So I think everyone should build a harness.

They went from selling shovels to all gold prospectors to stealing the information about the location of the gold so they could dig it out first.

We are all stupid enough to keep buying shovels from them because we think their shovels dig gold better and faster.

Am I to understand that this is essentially their form of social-platform ghosting instead of banning?

So they're not even going to tell you that the question you're asking is against their rules, they're just going to twist up your question and/or the answer somehow such that you waste your time essentially?

It seems like I ran into this EXACT same functionality from Claude many months ago when I was trying to ask it to research on the web and help me setup the ideal llama.cpp config for local llm inference.

Funny how lost it got through that relatively simple install when we had all of the documentation in the world (and a human dev with 20+ years experience guiding it along) to go by... and simultaneously it's debugging and building high level cryptography code in rust in the other terminal tab.

This is infuriating to learn.

This experience has made me feel like we have to create a community that moves AI from the mainframe era to the PC era quickly, or we will end up serfs.

I tried the same prompt on gemma4 and qwen 3.5 and Gemma consistently failed to call the multi line edit tool.

But it gets stuck in tool call loops, it seems like.

this is LLM, it's not like a science or something.

and they want me to pay $100+ a month to be their training?

i hope we can find morality again.

Come on guys, why can't everyone just be there for the good guy?

First, they want government to get involved and regulate frontier model development - even stop it completely.

Second, poisoning output of a model configured on the computers of millions of users goes way beyond protecting IP. That's malware.

• My most noticeable immediate jump was in how its frontend design was much more intentionally crafted, and delightful without feeling like 'AI vibe coded'; with better end-user usability too.

• In some internal agentic harnesses, it achieved better results with about half the tokens, making it cost the ~same as Opus 4.8 price-wise! The real price increase is less than 2x; with biggest differences in harder problems where Opus 4.8 struggles (or needs many turns).

• Part of the token efficiency improvements come from Fable doing more targeted and surgical diffs, with less non-necessary changes. This is great, because PRs often have less LoC changes for review. It writes more maintainable code without explicit human steering.

• For general conversation and assistant style use cases, didn’t really notice a difference vs 4.8.

• 1M context window, without increased pricing for long context is AWESOME. This is a massive win.

• The classifiers are super aggressive and sensitive and this does happen for very benign, non-security coding tasks. Fallbacks to 4.8 worked like a charm; but the filters are definitely super sensitive.

Overall, I would describe this as a step change and worthy of the "Claude 5" model name. It did take some time to understand the intelligence ceiling of this model; and even with an extended testing window I'm still discovering new things and often surprised (in a good way) by the model.

In Codex, GPT‑5.5 is available for Plus, Pro, Business, Enterprise, Edu, and Go plans with a 400K context window.

You can use Pro on the web if you’re on the Pro plan but not in Codex

It felt, at least for me, light an impressive step up. Opus 4.8 was already very thorough; but sadly verbose and ‘loopy’ when you push back on its plans. Fable is what I’d use all day if I could afford it!

I wonder how much of design capability improvements is related to our collective ability to recognize AI design tropes.

Look, I don't want to argue about something dumb like that, but you can give it basic instructions of what the UI should look like, how to group things, and an example image from a designer, and it will nail the result. If you don't think that's incredible, that's fine. I do.

Opus 4.7 made this a practical approach. 4.8 improved it. Fable 5 has improved it more.

Given the shit we've seen shipped by "top end companies" (all the way to Apple) I seriously doubt that. I'd say you're nitpicking from an artistic point of view or something.

so this is why claude talks like this, i was wondering where it was getting this verbal tick from.

I assume it might be a good barometer for generalised intelligence; esp in the visual space.

This seems like the pharmaceutical method of get them hooked on the drug with free samples, then once they can't live without it, raise the price. I'm not sure I want to start using Claude Fable on a max plan if it's just going to go away on June 23rd.

But maybe the more charitable reading is that they didn't have to offer this model at all on those plans and they are giving the standard free trial.

API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited

Limited "free" time is what game developers do if they want to stress test the infrastructure code until it breaks.

For the stuff I've thrown at it, that configuration has done a really great job. Including 70+KLOC go proxy with extensive test suite, some retro games, and more.

IMO the data from chats alone is worth $200B to Google.

Fable has literally refused to work on any of my problems (even those about fluid dynamics!) and just tells me that I'm violating anthropic's AUP. I've reached out to their support and don't expect to hear anything sensible back. One thing I do look forward to though is OpenAI offering an equivalent model but with less safeguards...

I dearly wish you could leverage the latest models to enhance your research.

A bit more context if you care: it's a meso-scale, physiological simulation environment of "particles" that carry nuclear spin, can move in 3D space, and (should they interact with each other or their environment) undergo chemical kinetics. The idea is to simulate molecules within e.g. organs or blood vessels within a person in an MRI scanner, with the motion of the particles dominated by the Navier Stokes equations, but here solved in a Lagrangian (rather than Eulerian) framework by smoothed particle hydrodynamics.

The fact that particles carry nuclear spin means that we can solve the (semiclassical) Bloch equations and by using a python plugin module import exactly the physical MRI scanner would do (in pulseq format) and be able to predict what signal the machine would record – e.g. there's a whole world of cardiac or neurological flow imaging work done in the context of nasty diseases like stroke or myocardial infarction – which has a bunch of physical artefacts behind it. I'm trying to make a simulation framework that can take in realistic patient geometries and act as a 'data generating process' because if we do it right the various physical artefacts that the machine records are reproduced, surprisingly accurately. Of course you also know the ground truth of where the particles are. I'm specifically interested in a weird technique (which I did my PhD in and you can read an article all about here: [0]) called dynamic nuclear polarisation, where specific spin states of molecules such as [1-13C]pyruvate are injected essentially out of thermodynamic equilibrium and act as short-lived tracers of metabolism – again highly altered in disease. The signal we record is a strong function of the physics of what you told the machine to do, the spatial constraints and environment of the patient's body, and the chemical kinetics of the patients' biochemistry (the latter two are usually what we're interested in).

Getting them to do chemistry as well as act as a "simple" tracer is more involved, because in the Lagrangian framework the number of particles is ≈ the spatial resolution of your simulation. That's fine if you're simulating water, but if you're simulating something that reacts concentration is not scale invariant (if you want to keep the interpretability of the rate constants). I've worked out an analytic set of scaling rules around this and fortunately for my application environments and length scales "it just works", completely by luck.

I've used Claude to port various SPH algorithms and boundary condition handling ideas (which are absolutely critical and highly not obvious – we have leaky walls in some places, and e.g. LCR / circuit theory models of the microcirculation to plug in) and it's been a godsend. But I'm running into its limitations constantly. It both confidently makes shit up, claims it is mathematically justified and when the resulting simulation explodes says "I apologise; I lied above" (!) or "I apologise; I am wrong" and I periodically have to yell at it to try to do something more productive.

The real hope is that this simulation environment would be both generally useful for basically anyone doing flow MRI, and help our basic scientific understanding of what we're measuring (the technique is in many hospitals!) but also be able to produce meaningful synthetic training data for image reconstruction algorithms later on. It'll end up permissively licensed (all of the "starting" codebases have compatible OSS licenses, and we're releasing our contributions similarly).

I really hoped that Fable would be better at this sort of work. Occasionally, relating to my work DNP [1], I have need to talk about proper nuclear physics and I have seen Opus's chat interface write a wall of text (e.g. talking about photonuclear reactions and cross section differences in millibarn) and then just delete it all. Support have told me that yes, I've hit the nuclear filter and, well, tough shit, basically.

I wrote a version of the above to them yesterday, and just got the most boilerplate response that I've yet to test:

    Thanks for reaching out to Anthropic Support.
   
       We're sorry to hear of the issue that you're running into with accessing Fable 5. I'm happy to say the issue has now been resolved and you should be able to access the model within Claude.

    I'll close this case out for now, but please feel free to reach back out to us here if you have any follow up questions or concerns or if you're still in need of assistance. We'll be happy to help.

[0] https://physicsworld.com/a/dynamic-nuclear-polarization-how-... [an "accessible" article] [1] https://www.science.org/doi/pdf/10.1126/sciadv.adz4334

>"it just works", completely by luck What does your validation function look like for this? Whenever stuff "just works" for me I get a little nervous until I determine why.

That's a whole separate long answer. I'm not a qualified doctor (and nor would I claim to be), but after a masters' degree in particle physics I moved into an explicitly interdisciplinary training programme that led to a doctorate and at other places in the country I did it in, a separate MPhil. During that initial year I spent a fair amount of time in the dissection room, learning anatomy, as well as most of the first three years (the foundational, preclinical part) of a medical degree combined into one (which contained lots of molecular biology, frankly). My final doctorate was between the departments of condensed matter physics (nominally my awarding institution), biochemistry, radiation oncology, and "the department of physiology anatomy and genetics", which is basically preclinical medicine. The people I work with are 50/50 recovering engineers or physicists, and qualified clinical medics who are trying to learn things like perturbation theory in their time off…

>"it just works", completely by luck What does your validation function look like for this? Whenever stuff "just works" for me I get a little nervous until I determine why.

Ah. I do know why: the relevant Damköhler numbers [0] are either very small (chemistry is much quicker than flow) or large (flow is much quicker than chemistry). So the approximations I am building in are justified and an awkward middle region is excluded; we also are only interested in small concentrations in a carrier fluid (e.g. blood, lymph) where the presence or absence of the species in question does not change its rheology.

I am lucky because we have evolved this way. If our circulatory system and its approach to metabolism was more similar to e.g. a reacting polymer foam ("can of expanding foam") which completely consumes its reactants as it goes, this implicit Lagrangian approach would likely not work.

[0] https://en.wikipedia.org/wiki/Damk%C3%B6hler_numbers

When GPT 4.5 launched, the gains compared to the model size didn't seem that great, leading some to believe that the only progress we'd see would come from RL.

This model certainly has quite a "substantial amount of post-training and fine-tuning", but it's also based on a new pretrain[1][3], which given the cost, indicate that it is in fact quite a bit larger than Opus 4.X.

[0] One of the early testers mentioned: "As far as I can tell from talking to people internally at Anthropic, there's nothing special about architecturally"[2]

[1] Section 1.1 in https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c3...

[2] https://youtu.be/GrdEid8H6H4?t=168

[3] There were rumors going around when Mythos was first announced that it was the first 10T parameter model, but I can't find a verifiable source for that number.

It turns out that having a text based interface for a text-trained model creates a very nice feedback loop.

Right now as we speak, people are generating text traces on anthropic and OpenAI servers that teach their models to do everything under the sun, text wise.

So people right now getting super mad at how dumb the model is when reverse-engineering a super complex function from binary, when they write “stop, you dumb robot, you are going wrong, go this way thank you very much” are actually leaving a lesson in the form of the "chat" text history.

Some may say that each bad word get us closer to ASI.

That and obviously the order of magnitude more efficient GPUS we got that allow for different tradeoffs at training time.

A typical session is the agent establishing a metrics and log baseline, creating the code, compiling, deploying, observing, fixing, redeploying, observing metrics, determining the outcome and commiting.

I really, really, don't look at the code anymore.

UPDATE:

so my point is: it won't have my stewarding the code anymore, but it will have the infrastructure (and ultimately the real world) providing feedback on the traces.

Maybe we need some form of long-term training. How long does the code that the AI wrote stick around before being rewritten.

I guess we can do this retroactively too if we could somehow tag AI-written lines of code in the VCS, then in a couple years we can check which parts lasted.

sorry. how do you know. i am so curious about where exactly gains are coming from but so hard to even get a little bit of insight.

i wish govt would fund these labs and make it free and opensource. way better investment than stupid overseas wars.

You have a false definition of "impossible." It would be true to say it could be challenging, given current political dysfunction, but it's not impossible.

> ...and even if they could, they would do it in a way that would get 90% frittered away to fraud and waste

Same with private business.

I'd prefer government funding, because there a greater number of important goals than the two or three the market is capable of optimizing for.

https://www.whitehouse.gov/presidential-actions/2025/11/laun...

Doctrine and propaganda can make someone that sure, and the thing they're sure about doesn't even have to be true.

> There's been massively successful government funded and run projects before. Soviets beat the Americans to space, after all.

Don't let facts get in the way of ideology!

Also the Americans subsequently beating the Soviets to the moon was the government literally allocating huge amounts of capital towards the literal trope-namer moonshot.

Apart from all the above: the fact that they are intentionally writing this (that they degrade frontier LLM dev, silently vs loudly for biology/cybersecurity) in the system card is interesting to say the least - especially just before IPO.

Notice that with this statement - that they're going to intentionally hobble the model for frontier LLM development - the general discussion has moved from, “Is the model actually that good?” to "they’re pulling the ladder up from behind them"

That's actually super smart - wonder if Mythos (or the next unreleased model) had a say in coming up with that strategy (if it's intentional). Also - having access to extremely capable models before anyone else - which they have by default - is a incredibly advantageous position to be in.

There's a quote from a METR report on page 52:

>We ran [Mythos 5] on 38 of our hardest software tasks, including tasks centered around R&D. [Mythos5] generally outperformed an early checkpoint of Claude Mythos Preview in these, including by succeeding on some tasks that had not been solved by any public model we have previously evaluated. However, we still observed the model occasionally failing to correctly interpret nuanced instructions in difficult tasks... Based on the available evidence, we believe [Mythos 5] is likely unable to fully and reliably automate R&D for frontier projects spanning multiple weeks. We believe that a better, more confident assessment would require more time, evaluations, and information from the model developer.

this is good news, right? right...?

- Opus 4.7 xhigh: 5.2%

- Opus 4.8 xhigh: 13.4%

- Fable 5 xhigh: 29.3%

Seems like a huge jump.

[1] https://cognition.ai/blog/frontier-code

1. That estimate could easily be wrong.

2. That estimate is, of course, usable in RL training. This isn't an inherently bad thing, and this is more or less what has improved coding models so much lately. But it does mean that other companies could and surely will do this sort of training, and Anthropic probably did too.

3. OSS maintainers are far from perfect, and there's an unfortunate uncanny valley-like effect in which a coding model can produce code that is just convincing enough to pass review even though it's actually totally wrong. I don't know whether this is a specific issue here.

prior bms relied mostly on unit tests or synthetic judges which are easily benchmaxxed, which leads to nobody trusting benchmarks

we need people manually checking the data for good code quality

this benchmark looks very good from the methodology. a cog researcher checking the data themselves is very high signal (not scaleable so don't take the benchmark as gospel, but directionally good)

TL;DR - they worked with OSS project maintainers to build tasks. They score models based on whether a PR is mergeable. All tasks are graded by a human researcher. SoTA models have hill-climbing to do which raises the bar and inspires confidence. I'd say it's legit.

[1]: https://x.com/cognition/status/2064061031912288715

Nobody would have 800+ billion reasons to lie by commission or omission here.

they aren't married to a particular lab, most of their usage is their in house model i believe

I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.

EDIT: Oh I see, this is the best link for pricing https://platform.claude.com/docs/en/about-claude/pricing

So the price is double across the board...

From their pricing page, Opus 4.8 costs $5 per million input tokens and $25 per million output tokens [1].

[1] https://platform.claude.com/docs/en/about-claude/models/over...

I would have expected Mythos to be much more expensive than just 2x current Opus (which is clearly cheaper to run than original Opus)

Input Price $10/M tokens

Output Price $50/M tokens

Cache Read $1/M tokens

Cache Write $12.50/M tokens

2x Claude Opus 4.8, same as Claude Opus 4.8 (Fast)

Frankly, not even Opus 4.8 would be enough of an incentive to use at that price range (enterprise-wise; would not even bat an eye as a consumer)

But - these $3k-$5k/month/engineer bills are going to start to get attention soon - only question is whether the response is to slow down on the $$$ spending or reduce the # of engineers.

whats the logic in claiming its a borked metric when everything listed is an anthropic model.

This kind of storytelling annoys me. Give us more facts, less narrative drama.

What matters is scale. Did it deploy a novel zero-day exploit to overcome a problem? That's alarming. Did it kill a disruptive process? Pretty normal troubleshooting step.

Some people seem to think that simply uttering these ideas on the Internet is harmful (in the "don't give it ideas!" way); but the MIRI types were expressing them pre-ChatGPT in an attempt to warn people, so there was really never any chance of keeping it out of the training data.

But it's also worth considering here just how awful AI security postures have been. The MIRI types used to speculate about how difficult it would be for AIs to social-engineer users into granting them irresponsible levels of agency. It turns out that they don't even have to try.

Okay, I'm going to start running a Bitcoin miner on your machine, and then use it to buy time on Digital Ocean.

I've written out my CLAUDE.md, and I'll use SSH to transfer my context to that other machine.

They are the only one crying out loud about how dangerous their models are and are presumably also training their models heavily to be "safe". And through that training itself, the model learns about the other side - how are you going to teach a model to be safe, without teaching it what's not safe?

Kung Fu Panda opening scene anyone? One often meet his fate on the path that he takes to avoid it - Master Oogway.

Very interesting. I am not sure this will comply with organizational policies and standards protocols (HIPPA etc.,)

Almost… basically they have unlimited power to decide what data is kept?

You can’t tell a judge who’s ordered you to retain something that you can’t because you said you wouldn’t.

Enterprise plans allow admins to set which models are allowed.

This feels more like working with a competent peer than ever. I won't use it once it's API-only, though. I don't mind guiding Opus as required and staying closer to the code. I can tell that Fable would lead to a lot more 'set and forget' programming which I'm still not fully comfortable with.

Regardless, this is cool. It's very fun to use. It was able to find legitimate issues with my work this week and we've made meaningful improvements. Opus can do this, but typically in much narrower contexts, and often with hallucinations or partial-errors. It needs to walk many things back or revise plans. So far that's not the case at all with Fable.

edit: I just realized I had Opus review the same work already. It missed everything Fable caught today. And it's actually worthwhile stuff to address. It's hard to say no to a model which demonstrably makes your code better, but... Those API prices will be brutal. Maybe a review here and there, I guess.

> Fable 5's safety measures flagged this message for cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Switched to Opus 4.8. Send feedback with /feedback or learn more: https://support.claude.com/en/articles/15363606

Seems like GPU drivers are cyber weapons of math destruction now.

Seriously, GPUs are a mess and keeping LLMs from helping us use them properly is practically a crime.

They kind of are, at least in the AI race.

> weapons of math destruction

lol. great, whether intentional or not.

The frontier labs now have every reason to hold back and sell only to their preferred trading partners. I don't really like the new arbiter-of-knowledge system we're barrelling toward.

● Bash(/tmp/run_ps.sh '& C:\rhombiq\d3d-probe.exe 2>&1 | Select-Object -First 4 | ForEach-Object { [Console]::Out.Write("$_`n") }' 2>/dev/null) ⎿ Adapter[0]: Qubes virtio-gpu WDDM 3D (dev) VendorId=0x1af4 DeviceId=0x1050 VRAM=8192MB Adapter[1]: Microsoft Basic Render Driver VendorId=0x1414 DeviceId=0x008c VRAM=0MB Adapter[2]: Microsoft Basic Render Driver VendorId=0x1414 DeviceId=0x008c VRAM=0MB

● Please run /login · API Error: 403 The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()

Brewed for 8m 35s

Continue please

● Your organization has disabled Claude subscription access for Claude Code · Use an Anthropic API key instead, or ask your admin to enable access

Seems like they locked by account.

* From today through June 22, Fable 5 is included on Pro, Max, Team, and seat-based Enterprise plans at no extra cost.

* On June 23, we’ll remove Fable 5 from those plans. Using it after that will require usage credits. If capacity allows, we’ll extend the included window.

* After this point—when sufficient capacity allows us to do so—we aim to restore Fable 5 as a standard part of subscription plans. We intend to do this as quickly as we can.

The "offer, then remove" aspect is a bit eyebrow-raising -- it feels like they are trying to get subscribers to switch to usage-based billing, which makes me wonder if we'll ever get it after that June 22nd window.

If they didn't announce it, you guys would be complaining about slowed progress.

If they didn't release it, you guys would be complaining about fake promises and marketing.

If they released it without limits, the complaints would be about slow responses and outages.

If they didn't add to susbcription plans, the complaints would be about phasing out subscriptions.

If they added to subscriptions with cost reflecting their resource availability, the complaints would be about how quickly it eats limits.

So they choose the middle ground of providing some initial access and assessing if they can satisfy demand, only to still be ignored and accused of trying to get users hooked?

We've already seen that they don't have enough compute, thus the deals with SpaceX for their GPUs. It's very reasonable that they just don't have the capacity to support the subscription userbase on this model.

I can recognize so much of the GPT/Codex generated code long after it gets merged (not by me).

Additionally, the time spent on every agent turn on GPT 5.5 is much longer compared to Claude Opus 4.8, which means iterating on the code takes a lot more patience, and there's a lot more nitpicks to pick when actually using GPT 5.5 to do software engineering.

Feels like GPT-style models are more geared on doing one-shot software vibing (and handling the vibe coded mixture) compared to Claude's focus on actual software maintenance. I got a GPT Pro sub for free and wanted to cancel my Claude subscription so much, but I still keep reaching Claude models a lot more. Frustrating.

this is the line I keep in Agents.md that helps me prevent Codex from playing smart

https://arxiv.org/abs/2602.10144

When a "person" that you don't view as a "real" person repeatedly does exactly what you just told it not to do (often amid false assurances it understands and will avoid doing so in the future), most people get angry.

Compare it to how the kind of people who treat children like property treat their kids, or other examples of keeping people as property.

who, or rather what, is being abused here exactly ?

You should see the abuse my motorbike gets. Poor thing.

We were reviewing reports of situations where the models failed to follow directions and there was a common thread of some where when the operator got the model to acknowledge the rule breach, it quoted back something that included swearing.

I don’t have the data to truely look into it, but I did give the instruction to my engineers to avoid it as a “might be a problem”.

But I avoid unnecessary emotion in my prompts because I don't want potentially distracting activations. Kind of like communicating with humans.

> impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts.

Unless the mechanism is understood, my assumption is that this is a moving target.

How so? Plenty of swearing in lots of training data, especially older code, e.g. in Linux.

Bonus points if you find yourself actually saying it out loud while typing it.

I have used the word "shenanigans" way more in a couple of years of agentic coding than in 30 years of writing code with humans.

ai llm are doing what i tell them to.

if you’re building something meaningful (in my case a platform used by many people across many companies) you want to ensure you

1. have actual systems engineering and architecture in mind that you want the models to

2. implement based on what you tell it to do

when i was just telling the models what i want done without doing due diligence it would go and do some moronic implementation that was awful. mid input = mid output

these days i just maintain specifications documents and the AI follows everything i tell it to in that document. so when i tell it to dos one thing, the result is made following those architecture specs.

i have code that is single resp, modular, easy to extend and test.

i would ballpark 95% of the time i get what i asked for.

sometimes it tries to be clever in cases that weren’t covered in my arch specs. in those 5% of cases i go and update my specs.

source: used billions of tokens worth to build something actually in production across both mobile platforms and web, deployed on my own cloud infra. i use codex mainly. some claude.

But Claude models seem to be better at long term problems or more ambiguous problems.

I'm curious as to what the primary benefit here. Are there secret improvements in training? There hasn't been much in fundamental model architecture, I don't think. What about harnesses? I wonder what's pushing the AI. It seems like harnesses is the main thing pushing AI ever since CoT.

I think the end game is routed model usage and SLMs. I think Apple is going to prove this in the consumer space pretty handily and I'm curious how the Android ecosystem responds since the hardware is considerably lacking in model performance. I think Apple has a huge opportunity here, as much as I don't like their current ecosystem of walled garden. They did position themselves very well with ARM and custom chips for their hardware. Hopefully the broader ecosystem of ARM and Linux are able to make some headway and we see a more formalized, and broadly accepted, architecture to capitalize on.

I’m sure you could put something similar together with a bunch of duct tape and 2 weeks of effort, but it won’t work nearly as nicely nor out of the box. so…what am i missing?

My company has an agreement with the big providers and while i'm pretty sure they think about how to get budget back, its an competitive advantage and normal people will not learn different model behaviours.

At least for now.

Regardless of what others are doing, US labs here are just rushing to IPO. It's NOT a sign of confidence.

It's the equivalent of saying you have confidence in SpaceX making revenue by renting out their data center (instead of their AI making bank).

On the same note. if spacex is doing datacenters on earth successfully what's wrong with that? They rented cloud infra to a #2 or #3 provider in the world after < 2 years in business. It's a success, no?

If you get hired as a staff engineer and do the work of a junior, what's wrong with that?

Clearly xAI (now part of spaceX) did not raise funds to be a data center. The margins are way different. There are plenty of recent IPOs in that area that are worth at most billions not trillions.

> going to IPO is a sign of confidence , you need to report a lot of things, that private companies don't.

This isn't going to IPO. This is rushing to IPO. It is a sign of confidence that the market or wider environment might crash soon so we need the liquidity now.

> This is an exact reason chinese labs do not rush to go public.

Maybe or maybe not. If you are referring to Chinese labs - both the Hong Kong and China stock market are way weaker than Nasdaq. It's not comparable. Check all the recent Hong Kong IPOs that have tanked.

So no, reason not to might just be: no money in it.

There are huge numbers of users (myself included) that do have an exact idea of what inference costs are - on open models. We can buy tokens from 3rd parties that have no motivation to subsidize our use. That's to say, there's a fair marketplace[1] and we're hanging out there.

If you want to say "I don't think anyone has a firm grasp on actual inference costs on these proprietary/closed models", then I could agree with that.

[1]: https://openrouter.ai/rankings#leaderboard

China subsidizes strategic industries, and they have heavily done so with AI. And DeepSeek specifically has said they have no commercialization plans.

For example: https://www.boc.cn/aboutboc/bi1/202501/t20250123_25254674.ht...

It’s generally established that Anthropic/OpenAI are going for all out performance with big VC dollars at the expense of efficiency and China has geopolitically limited compute and an inventive to compete on value per dollar.

Why not? Hetzner charges WAY less than AWS too. Can you not believe that?

Both. They are charging the most they can get away with and that amount is still heavily subsidized by VC capital.

We know roughly how much these companies spend and what their revenues are. Based on that, they'd have to more than double revenue (without spending more money) just to stay even, and that's not good enough given how deep in the hole they are.

> OpenAI and Anthropic are heavily subsidizing their inference -- no wait, they are charging the most they can get away with before going public. Where is the truth?

Both are true. I mean, I'd be willing to spend a bit more than I do now, but not more than double, and neither are most companies. The company I work for is currently investigating how to reduce LLM spend, not looking to spend more.

Now that 200USD subscription starts to feel cheap...

I haven't gotten close to this either before, but now we wanted to move fast because this branch gets conflicts all the time and we want to get over with the migration asap.

And don't get me wrong. Opus did an absolutely horrible job at first, second and third round in this task. You really needed to steer it to get to the right solution.

And now Fable is out. And its first round of code reviews for this huge PR was definitely worth the money too...

Don't think that I'm just shrugging to that number. I see it every day, and I don't like that it's in the thousands now. But for people paying the 100 or 200 dollar plans, I'm not super sure if you will be able to use them in the future if the token price is in the thousands for a bit bigger task...

If I'd pay this from my own pocket, I'd definitely go with DeepSeek or local models and figure it out how to make the best use of them.

I wonder what this question really means? Anthropic is useless if you don't know what to do with it. It's very useful if you do, and you can guide it to do the right things. Yes, it will for sure reduce the amount of people we need to hire. But we are always looking for hires who know what they do and can utilize agents to be faster.

But if you think about how long employer is willing to pay 10-20k per month per seat for Anthropic? I can't see this to be feasible and it will have to end at some point.

It's worth it, and I can afford it, but I am not really the right type of user for token-based usage. It's all for personal and free work.

Unfortunately, that doesn't work within a single session. The K-V cache of a model is intertwined with the model's configuration. Switching models invalidates the cache, meaning everything up to the point of the switchover is processed like a new, uncached input token.

Per Anthropic's pricing doc, an Opus 4.8 cache hit costs 50¢/MTok, while Haiku costs $1/MTok for uncached input.

Model selection works best if sessions are short and self-contained, particularly if the first few interactions can reliably classify the model need. That probably covers most 'support chatbot' use-cases, but it doesn't describe the kinds of heavy agentic automation that really chews through token budgets.

I don't think this is true if you simply quantize the model or run it with fewer active experts? The underlying weights would stay the same. You could also play further tricks with skipping some of the model's middle layers outright, which works surprisingly well due to how skip connections are used.

Most AI companies are just testing the waters with paid tiers right now, their greatest fear with increased pricing is folks reverting back to wikipedia, stack-overflow and other public domain organic activity buzzing back to life; that will kill any RoI potential in LLMs forever. They're playing the wait game instead, observing how the digital sphere reacts to every little increase in price.

If that weren't the case, they'd be pricing at lucrative premiums already and even gotten away in short-term considering the increased dependency in the enterprise world. But that'd be like killing for the golden egg too soon and losing all long-term potential.

Once the folks are so addicted to LLMs that even writing a hello world program sounds like a nightmare and coming up with an article draft feels like reinventing Egyptian glyphs, that's when the real pricing hammer will come.

Even if subscriptions are locally profitable (i. e., the cost of the subscription covers the cost of inference), they're still subsidized because they don't cover training and running the company; otherwise, these companies would be profitable.

Take a look at China for example - they have no access to NVIDIA, so they're trying to build their own hardware, they have no unlimited funding, so they try to optimize things.

And Anthropic is complete opposite of that - if NVIDIA were to triple their prices tomorrow, Anthropic would still pay them.

In the end, either we all somehow go mad and start paying Anthropic tens of thousands of dollars per month so support this madness, or we will go with whoever isn't lighting cash on fire.

Not true. Stop following US media spam if needed.

1. Very recently, the US did close a loophole on sanctions that allowed Chinese companies to use NVIDIA hardware outside of China i.e. before that was closed they all had access. The trick was train outside, do adjustments, ship the disks back and use non-NVIDIA in China, but at least the training and endpoints not hosted in China could all use NVIDIA.

2. There's been plenty of reports including fines and bans e.g. to Supermicro on smuggling NVIDIA hardware to China. I doubt it has been stopped. You can't catch everyone.

Granted, it could still mean that Anthropic just chooses to lose money - but that's Anthropic's choice.

DeepSeek has proven that inference can be much, much cheaper than what Anthropic advertises on their API rates page.

Then the cost is being subsidized by investor capital, but it is still subsidized.

So they are profitable?

I think you are mismatching accounting terms.

You can't say the 'subscriptions' are profitable without accounting for the cost of making the model that is the source of the subscription.

They are heavily subsidized by the shareholders. Investing, running at a loss, with hope of some future profitability.

If saner factory can sell you the same tool at a fraction of the cost of a gold plated factory, your choice is going to be obvious.

Having said that, I found the cloud dev environments slow to the point where I wasn’t sure if it had frozen, so I never looked back.

Though the day is coming when there’s no distinguishing, I’m sure.

Just like how changing Kennedy Center letterhead to Trump Kennedy Center for a year didn't actually legally rename it.

Once a case with sufficient standing got in front of a judge it reverted to the actual legal name on the basis that only Congress can change the statutorily defined name.

For an admin so obsessed with legal names instead of chosen ones, you’d think they’d be less hypocritical.

I'm doing basic web development here utilizing animejs. Nothing too complicated (mostly saving time doing the scaffolding, still write the bulk of animations manually).

Truly believe that American companies are going to get completely curb stomped by China due to greed, ineptitude, and violating the social contract.

Deepseek V4 Flash is suprisingly capable and insanely cheap. It takes so much to get the session cost to get to $0.01.

I agree with you on pricing, but what do you mean by this?

Why aren't corporations doing more to help workers with childcare? Why aren't they doing more profit sharing with workers? Why aren't they encouraging unions or sectorial bargaining? Why isn't the government mandating any of this?

Americans very rarely benefit when US corporations do well. That needs to change. No one benefits if Meta continues making billions in profit every quarter while society suffers from isolation, depression, suicide, and scams from their services. Americans don't benefit if health insurance companies are making massive profits while they can't afford deductibles.

Our society has been setup to simply extract wealth in all facets of life. That's a sick society and it needs to change.

I'm not saying China does this better, in fact China has some of the worse worker rights out of all the industrialized countries; but at least American consumers would benefit from cheaper higher quality Chinese goods. The world would likely benefit too if America got off the cold war hype train that did nothing to benefit humanity outside of those making weapon systems.

The AI companies sure are a brilliant example of corporations needing to do more to help their employees pay for childcare.

I am on the $100 Max plan.

I do wonder if you switched models mid-session, you would have lost all your cache. Reloading the context into cache can really eat through your usage.

I had it analyze a project I was working on with Opus 4.8, and it blew through 23% of my session limit in one go. Does not portend well for my budget.

> Fable 5's safety measures flagged this message for cybersecurity or biology topics.

> They may flag safe, normal content as well.

> These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them.

Here are the results of the agentic code review session:

  ┌──────────────────────────┬───────────────┬────────────────┐
  │          Agent           │ Fable 5 turns │ Opus 4.8 turns │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ values                   │ 134           │ 0              │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ data-intrinsics          │ 104           │ 0              │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ tools-tests-build        │ 81            │ 0              │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ core-intrinsics (failed) │ 25            │ 0              │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ system-memory            │ 44            │ 20             │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ reader-modules           │ 104           │ 25             │
  ├──────────────────────────┼───────────────┼────────────────┤
  │ linux-startup            │ 95            │ 15             │
  └──────────────────────────┴───────────────┴────────────────┘

Or is it just not allowed to find bugs? Or it's only allowed to tell you bugs that don't pose a security risk?

Seems that way. "Security" was never part of the prompt. It was something like:

> Hello, Fable! Can you give me a complete code review of my lone lisp project? Opus has already done extensive code review. I'm curious to see what you say.

Result was the table above.

They also, FWIW, say that they've instituted new policies on their end such as logging any human access to the stored data and automated deletion after 30 days in "most" cases (with another link to a document detailing that further).

Assuming this isn't just a supply issue on their side, nothing says "ethical AI" like only allowing mega corporations to use it through cost barriers.

How many government sanctioned school bombings does it take for them to quit working with said government? For now we know that number is somewhere between infinity and 1.

The question of collaboration with USG is a much more complex one, but is not the one raised above.

Edit: I'll also add that I doubt any AI-doom people "trust" Anthropic per se. The entire angle of questioning – again – misunderstands the AI-doom argument. You appear to think that if companies behave unethically, they cannot be trusted and they will not produce good outcomes, inversely: if they behave ethically, they can be trusted, and they will produce good outcomes.

Any competent AI-doomer would argue that ethics or trust are essentially irrelevant.

The entire problem is that people can act totally reasonably, even ethically, and this is not a guarantee of good outcomes. Situations can be created in which completely ethical, reasonable behavior actually produces a bad outcome. You do not need to assume people are bad in order to produce a bad outcome, and inversely you cannot assume that you will get a good outcome from good people.

"Arms races" are one class of situations that often have this characteristic. "Bureaucracy" is another class that we encounter a lot in daily life. There's a lot of them!

Talk about a strawman!

This "simple" fact needs quite a bit of additional context and work. Making grandiose ethical claims like this can be countered with other grandiose claims such as the fact that there is no ethical existence under communism or socialism.

The fact that there is no ethical consumption under capitalism is not material to whether or not ethical existence is possible under communism or socialism. In order to survive in a capitalist society, one inherently has to make choices that require trade-offs, and those trade-offs are burdened by a history of decisions made not just by the people alive today, but our ancestors as well. Does that mean I walk around chanting "Reparations", "Land-back", or other calls to action? No, but I do acknowledge that there are unresolved issues and as a Canadian, I know we need to do more to resolve treaty issues, and environmental issues, and system discrimination. I also know that Americans need to do better to address systemic discrimination and many, many other issues. It also doesn't mean I want to give back my house, or give away all of my possessions. It just means I try to make good choices and support businesses and people that are open about the trade-offs they make and try to engage as ethically as possible.

Acknowledging those facts doesn't absolve us of responsibility, it's a framework that allows folks concerned about whether or not they are doing the right thing to accept the trade-offs that they choose to make and be responsible and accountable for those choices to themselves or their communities.

We live in a world with scarce resources. It's possible that with a foundational redesign of the global economy, and the requisite authoritarian government that would be required to force such a redesign, we could eliminate food scarcity, solve energy scarcity, and make sure that everyone has a place to live. Those trade-offs are probably not worth the ethical cost in political and physical violence required to accomplish it. We have seen the trade-offs that happen when the powerful are able to exploit communist or socialist governments. We are seeing the "late stage capitalism" impacts of allowing the powerful to exploit capitalism in democratic societies. Acknowledging that the current capitalist system has lead to the greatest prosperity for the upper echelon (financially) of humanity, and a dramatic reduction in global poverty shouldn't obscure the reality that much of that wealth comes from exploitation of people and the environment.

It's a huge problem to unwind, and we can't let the burden of every choice that we make stop us from trying to do better, but we (as in society in general) can't do better if we don't at least acknowledge the compromises we are making along the way, and try to plan to fix it in the future.

Probably a topic better suited to beer and a pub setting than HN though :P

I don't believe that this is a fact. How are you demonstrating that this is a fact?

When you talk about things like reparations or "land back" you're already cargo-culting in concepts and ideas that themselves need to be fleshed out in order to make a subsequent claim that a specific economic system is unethical. Someone can just argue all economic systems are unethical, how are you going to defend against that? And can you pay reparations for example without going back in all of human history and finding all cases of injustices and then tallying it up? Why pick an arbitrary point in time? Better yet, why not start in countries where slavery still exists instead of focusing on the west which led the world in abolishing slavery and created concepts such as universal human rights.

Even with respect to "eliminating food scarcity" - eliminate in what sense? All olive groves and grapevines and rice farms have to be destroyed and rebuilt to only build certain foods?

Dabbling in communism or other inhumane and authoritarian governmental systems is extremely dangerous and in the same vein of extraordinary claims required extraordinary evidence, suggesting as you did creating an authoritarian government to create a utopia is precisely the same project of suffering and death that mass murderers throughout history have undertaken to abject failure, and thus, you need some incredible amount of evidence and theory to be able to even fairly suggest going down this path.

I am not going to do the work of gathering the evidence for you, and I don't think this is the right venue for a debate on the topic.

If you don't have evidence I think it's mature of you to admit that and applaud you in doing so. We all like to just talk and don't have to always provide evidence for every citation or what not and it's fair to just say hey I'm just making this up and it requires further discussion.

(I’m highly confident open models will eventually achieve a similar performance benchmark with distillation over time)

ZIRP (zero interest rate policy) is over, software engineers no longer call the shots now that there isn’t vast amounts of capital chasing yield, and that capital bidding up salaries and keeping the labor market for engineers tight.

If you are x more productive with generative AI, very shortly you are going to have to prove it with a token budget (or, if you’re lucky, an org willing to spend for on prem hardware for capped token cost, fixed capex vs uncapped opex).

The comparison is not SWE vs SWE with AI. It is SWE vs SWE with AI with a constrained token budget ($x/month) delivering the same value at the same or lower cost. If you cannot prove that you are wildly (vs marginally) more productive with the AI, why would they pay for it? Prove it.

https://abhishek-shankar.com/posts/ai-coding-bill-headcount-...

> That is the real content of the Uber story, and it is why filing it under "budgeting discipline" misses what is actually unfolding across half the engineering organizations in the country right now. They ran the same experiment Uber ran, most of them without Uber's $3.4 billion R&D cushion to absorb the surprise, and almost none of them having modeled the heavy-user tail or instrumented the gap between tokens consumed and value shipped. The reckoning will arrive for each of them on their own fiscal calendar, and the first instinct will be the wrong one. The tool is too good to abandon, the bill is too large to absorb, and the only durable resolution runs through a question the entire rollout was designed to defer.

> You cannot get labor-replacement economics out of a tool you deployed as a labor supplement, and the bill comes due before anyone is willing to admit which one they actually bought.

Why wouldn't Anthropic just wait until people start subscribing, do some kind of marketing push, or obtain some kind of other sustainable revenue stream, before they go IPO? I wonder if they see the writing on the wall with all of this and want to cash out as quickly as possible?

Specifically they need businesses that fired people and adapted their business to the products, so when the unsubsidized costs hit the businesses are forced to eat the true costs.

Yes they can't afford to give the products for free, but what is essentially happening with AI services is economic dumping, keep costs artificially low to get people to fire everybody, and then Jack the rates once they have Monopoly control

I agree. They need addicts, but they are high on their own supply and everyone else can see the danger in getting hooked.

I just use dumb and fast models now. I'm more engaged. I think that the higher the quality of the model, the more you tend to vibe with it, and then the more hallucinations you then miss. I'm not sure which is more productive, but I definitely burn out faster the more I vibe. At some point you're spending your time on forums, discord, or youtube instead of engaged with what you're building. Or you yak shave about your tooling and end up creating the 600th multi-agent gastown harness and blowing thousands of dollars on tokens to create it only to discover it's too expense to actually use.

https://cursor.com/evals

why do they have capacity now that they wont in a few weeks?

Opus 4.8 produces output in 15 minutes that is 3-4 hours of my work away from output that used to take me 40ish hours (a solid week of dedicated effort).

Last year(-ish, maybe it was 18 months, I forget when the jump happened), the frontier models couldn't touch this work. The output looked like a hardworking intern on their first day. Nice formatting, decent volume of words, but no understanding.

So it might work if it turns out to be a substantial leap in capability.

They'll probably tighten the quotas to reign in whales though.

Realistically I think Anthropic just has insane demand but finite capacity to run models, and Fable will just make them more money if they dedicate it to API pricing. I suspect the goal here is something like: get individual engineers/PMs on their personal plans to taste Fable and then go to their meetings and say "Yes doubling the price of every single input/output token is a good idea, boss".

The only reason why I pay $200 is because LLM's errors costs me that much, at worst. If "make no error" starts working - sure. But surely, unless you have millions of dollars of cash to burn, a coin flip that costs $5000 is an insane idea?

Going PAYG only will effectively take these tools away from a huge amount of people and accelerate the push for local LLMs.

OTOH, accelerating the push for local LLMs would also be fine with me.

The AI landscape is changing rapidly, and with Apple announcing the option to change the AI backend, and potential requirements enable AI choices as well, similar to EU browser choice requirements (this is more reading tea leaves than any actual requirements I am aware of). The new OS changes coming to support Googlebook, and deep Copilot/AI integration into Windows will make maintaining user facing subscriptions essential for independent model developers like OpenAI, Anthropic, and Mistal to remain relevant longer term.

If the don't maintain that relevance there is increasing likelihood that they will get consumed by other companies whether it's Apple, Microsoft or Google to form a foundation for their OS, or other cloud providers.

It's kind of annoying not getting access to the primo model and paying 200 bucks a month. I understand 200 bucks a month is basically nothing though.

Like I don't totally understand why they'd let me have it for a couple weeks and then take it away and say I can have it but I have to pay retail and retail is like $1,000 a day.

It's better to have loved and lost than to have never loved at all??

As a consumer I can choose to buy subscriptions to a range of things, including $5 droplets or VMs on a broad range of cloud hosting providers. I can even buy cheap bare metal at a bunch of providers at an affordable retail rate.

I can also buy "unlimited" AI packages that will be optimized to fit the cost model from a variety of services, with different impacts, such as rolling outages when I consume a daily or hourly allotment.

Right now VC and the investor class are subsidizing the rapid evolution of the services and availability, but that VC is running out. In more traditional economies, AI would have developed and rolled out more slowly, and through metered subscriptions, with the eventual rolling out of "unlimited" packages like telephone, internet, or cell services once the market became commoditized.

We have seen a big inversion of that with the race to "win" AI marketshare. Now the true cost is being exposed, and the most competitive and capable models are hideously expensive to operate, so it makes sense that we are moving to metered billing for a utility service. If you want gas, you can buy regular or premium. If you have a premium car you definitely want the premium, but for most people regular is good.

Give it a couple of years, and the survivors will settle around fairly industry standard models of consumer grade services, pro-sumer accounts, and business/enterprise models.

Things are still shaking out, but I get the sadness. Luckily I work at a big tech company who is banging the drum on doing experimentation so I use my prosumer claude pro and other accounts at home for hobby stuff, and save my heavy lifting and potentially experimentation for work :P

The newer models are smarter but really ficklle and hard to get meaningful work out of

4.6 was a workhorse

I think they might be hitting a point where subsidizing the expensive models for subscriptions makes less and less sense.

With Opus 4.X, last month I paid 100 USD for the Max subscription and got a token equivalent of 4.1k USD.

I imagine that Fable is more expensive to run.

Probably all about the IPO.

It's the same exact speed as opus >=4.5, sonnet 4.5, and twice the speed of opus <=4.1

It must have about the same active parameters, or else its a larger model running in turbo mode (smaller batches) and being heavily subsidized for some reason. But given most of the benchmarks are within 5% I doubt it is a much larger model. Most perplexing.

Do we know this? I’ve seen evidence they lose money on heavy users. But so do gyms.

Most gyms sell more subscriptions than they can fit under their roof at one time. If a gym only sells to heavy users, it will either be constantly turning members away or have to buy more equipment. Its equipment will wear off faster. Depending on amenities, it will go through towels, soap, water, et cetera faster, too.

Unless they're really, seriously wasteful with the soap.. there's no chance a gym is losing money on a heavy user

Right now all these AI subscriptions are priced like Planet Fitness, but they're used like Equinox. They're hoping that the new a la carte offerings will move their pricing more in that direction as well.

US gyms might be vast warehouses but in the UK, most only have a couple of benches, couple of cages, one set of db per denomination above 20kg etc. They require working-in and consideration for others.

A couple of unapproachable "heavy users" doing 3 hour sessions across peak hours can ruin the workout for dozens of paying members needing a few min per station for ~5 sets.

It might also be a euphemism for "dickhead" who also tend to be "heavy users". Those that damage, hoard and don't share equipment and repel other customers on many levels besides - threatening, lecherous, loud and smelly.

Doesn't even need malicious intent - can be weirdo bores, forever talking at victims while doing a routine that makes absolutely no sense besides camping on equipment for half a day... 100 sets of incline press 7 days a week... what are you even doing to yourself fella?

Where?

What I wonder however is if these tools will become something I use at work only. $100/month is already a massive stretch budget wise. If these models keep devouring tokens there’s no way I’d get the same usage time out of them for $100 in usage credits.

I just don’t think I’d use them much at all at home.

The step-up in intelligence looks massive (we'll see in practice), but the price is getting to a point where it's making me question if it's even worth giving it a try.

Good competitors will probably be out soon, which should level the playing field. I am more excited about that, just the fact that they showed that such an improvement is possible. I'm okay waiting a bit longer for this to become attainable for plebs like me.

Kind of like billing a programmer by the hour.

Perhaps not that close to US salaries, but those are inflated to hell. Worldwide senior engineers and scientists have salaries just about an order of magnitude away from AI subscriptions that you can use most of the day every day.

As annoyed as I am about this move, I get it. Users flood the newest, best model whether they really need it or not, and are efficient at using their entire quota. They've had so much trouble reigning in subscription usage it makes sense.

...

Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user."

[edit] -- I see that this comes from the system card -- dang merged the comments from the other discussion so that explains the confusion.

If you rely on this as a core part of your business/profession, you will be at their mercy and subject to whatever whims or challenges they have.

> Fable 5 · Most capable for your hardest and longest-running tasks · Uses your limits ~2× faster than Opus

Pay-as-you go isn't a common thing in SaaS. For example, except for AWS SES, all email providers are bulk-subscription based.

Sounds like "bait and wait".

If you think about it, the more people pay for these new and more resource hungry models, the longer it takes for them to become no extra cost and the longer it takes the more people are tempted to pay extra.

If you have good expertise in a domain and access to cheaper models, you may still be more skilled than someone without expertise but a lot of money to bruteforce the problems using SOTA LLMs.

Of course, they are a casino as well giving you free spins at the wheel with their new Fable machine, and it is done on purpose.

Once there freebies have expired, many of its users will begin to gamble more on the new casino machine and will realize that it is expensive.

The ramifications go beyond the individual which is why I assume they mentioned it. They don’t need to use it/not use it for it to have interesting implications.

Is it nice we get the trial? Sure. Is it also a common play in the playbook of tech companies? Yes.

Anthropic does not care about us and isn't going to talk to you either and will extract from you as much as possible.

The true answer is local models.

Seems like a concerted and distributed effort from the entire Anthropic team every time to get this on top of HN.

It happens for every single Anthropic release. Then I try it on real dev and the result is laughably bad. Except in design where it has been doing a decent job for a while. I am not a designer and my bar is pretty low.

So unless you have unlimited tokens it's better to learn frontend

Fable 5 has been a major improvement in high-level reasoning, like taking a plan file that has been optimized to the point where neither Opus nor Codex can find anything to change about it (neither in direction nor impl-detail), and Fable 5 will find high-level directional simplifications and pivots, or it will consider the best pivots itself and explain why it rejected them in favor of the plan's direction.

It's so expensive though. A single review of a plan file with Fable 5 (xhigh effort) will use 2-3% of my hourly limit on a $200/mo plan.

I think my new workflow is to generate the initial plan with Opus 4.8 (max effort), get Fable 5 (xhigh) to review it for directional feedback, then start the Opus<->Codex revision loop from there.

Fable 5 is 2x the cost per token of Opus 4.8, and it's much less work to review a plan than generate one.

Edit: It did correctly identify that transparent huge pages were off in its sandboxed environment and that enabling it was helpful, so that's nice. It also noticed that we skip THP on a certain less used path.

More importantly, I'm finding that the code that it produces for its experiments is a lot cleaner than what I'd expect out of Opus; there's fewer useless comments and it's more surgical and readable. I wonder if that explains the increased scores on benchmarks measuring mergability.

I understand that moving the goalpost every release is unfair, but it's similarly concerning to consider that people were letting GPT 4.X vibe code and ship entire products.

Stockfish does use neural nets but they are tiny, on the order of 10M params. Frontier LLMs are probably 100k or 1M times larger than that.

Someone trying to solve similar problems will have similar results if the "silent failure" applies consistently in aggregate. So, this is the model's performance.

My theory is that Anthropic are banking on being the top model when the race to IPO finally reaches the finish line, and to do that they need to have the top model but not let any competitors see it or derive from it to have a comparable model in the market.

Fable is their way of showing the public "the model does exist but in a mode that makes it harder/impossible for competitors to derive a comparable model from results.

The fable part appears to be that it's affordable by mere mortals. Anthropic support told me "too bad" when I requested a refund.

If this was a step change, e.g a Opus 5, I'd be pleased, it's definitely an upgrade on some work, but it's nothing like anthropics apocalyptical marketing seemed to suggest

How is this half-way down the page? To me it's the headline.

The rate limiting steps are generally testing, or characterizing. Not designing protein binders.

You would think he is churning our cancer drugs or something if you read his comments

I like separating the art from the artist in cases like this; he's clearly made very cool things in the past, but that doesn't mean he's perfect.

Fable 5 default: https://gist.github.com/simonw/036bee5a703e7ec84e34efa974438...

Opus 4.8 (the "max" one is closest to Fable): https://simonwillison.net/2026/May/28/claude-opus-4-8/#and-s...

Now here are the Fable pelicans for all five of the thinking effort levels - low, medium, high, xhigh, max: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Low used 25 input, 1,929 output - 9.67 cents: https://www.llm-prices.com/#it=25&ot=1929&sel=claude-fable-5

Max used 25 input, 14,430 output - 72.175 cents! https://www.llm-prices.com/#it=25&ot=14430&sel=claude-fable-...

When it started, comparing the progress between models was mildly interesting but everyone (including Simon) acknowledges it certainly leaked into the training data long ago.

> you still see improvements

This is expected if they are training their models on it, right?

> objectively-bad results

Keen to learn when this has been the case, i.e. across version increments in major models.

I've been enjoying seeing how the quality of individual models differ based on the amount of reasoning effort you give them. If they were baking an a good pelican you wouldn't expect them to differ so much.

(Google Gemini are the only lab that have very clearly paid attention to the quality of SVG animals-riding-vehicles, see their announcement for Gemini 3.1: https://twitter.com/JeffDean/status/2024525132266688757 )

that reply never failed to come it's basically a meme at this point

Clearly at this point they are part of the training data.

They even all look sort of ish the same. Daytime, colors,...

I know because I too had this initial take; however, upon analysis, it is not sound.

I agree as well that he writes many interesting things.

Fun at first, seems disingenuous now. A site funnel

well done anthropic.

Decided the best way to test this was to throw it a really meaty bone: a bug in lifecycle management of Chrome processes on Windows 10. Within the code-base I had developed workarounds over time with Sonnet and Opus, and while those reliably mitigated the problems, it always felt like a clutch and had some performance overhead as well as isolation requirements I would rather not have to take forward.

In comes Fable. Rather than examining the code base, and test a few fixes, Fable sets up an entire testing laboratory inclusive its own controllable webserver, fully instrumented to observe both Python as well as the whole OS kernel process environment, develops a suit of error reproduction tests, confirms the problem and the circumstances under which they reproduce, deep dives into the sources of project dependencies to look for the root cause(s), identifies these and confirms those hypothesis with further experiments. Looks for potential fixes in the later releases of the project where the bug originates, confirms this is not fixed, explores the documentation of said project to find other usage patters, expands its test suit to investigate these alternatives, confirms by crosschecking the source and running further tests that these alternatives do not fully solve the root problem, does a comparative experimental analysis of 3 different styles for using the project, checks the stated roadmap and developer activity in the commit history, recommends a switch to a different pattern that still requires a few of the process management workarounds (I told it not to patch external component), but that significantly simplifies the code-base ...

This is going to be a good 2 weeks, but what happens after? I can't afford this on a per token basis for my own projects.

P.S. An yes, midway the final implementation stretch I got the "Fable 5's safety measures flagged this message for cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we're working to refine them. Switched to Opus 4.8. Send feedback with /feedback or learn more"

Opus managed to finish the implementation, but they need to work on that false positive rate.

It’s interesting these companies have trained us to think that disruptive intelligence should be affordable to laypersons.

What will happen after two weeks is that people and companies with means who can afford it will get it, and folks without means won’t.

[1] https://support.claude.com/en/articles/15425996-data-retenti...

This applies even with API usage through third-party inference providers (e.g. AWS' Bedrock and GCP's Vertex) or with a zero-day data retention agreement in place.

I understand the reasoning for doing this, but I don't love the precedent that it sets.

A customer could sign a ZDR agreement with Anthropic, and their API usage wouldn't be retained for even a day. That's no longer possible.

These "karma" points are made up and are virtually worthless anyway.

From the model card:

In light of the ability of recent models to accelerate their own development, we've implemented new interventions that limit Claude's effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design. Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms. Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user.

Might be worth going back and taking a harder look at what I was asking it about if it somehow triggered a “forbidden knowledge” alert. Or maybe it was just a random bug.

Oh man all of those runaway infrastructure buildouts by our agents trying to achieve singularity...

Just say you don't want to lower the bar for others to compete

This seems so wide reaching if it's catching simple things like explaining a paper. Does this also refuse to help with any already developed training pipelines?

I can kind of understand the generation of synthetic data, but nerfing the assistance of training pipelines just seems like a really shitty thing to do.

It is not just biology but is defaulting back to 4.8 for me on time series/information transfer techniques that happen to mostly have papers using the technique on neural data. Other information transfer techniques are perfectly fine, even cutting edge ones, but this one happens to be new and happens to only be discussed in terms of neural data so that is a no go.

With that said, I think it is absolutely awesome. The usage is really not bad at all compared to what I was expecting.

Yeah... We need open models so we don't have that BS.

Fun times when “safety” means both the safety of mankind, and also the safety of revenues

Your priorities are not everyone else's priorities. The people concerned about AI extinction risk list those as three of their biggest priorities for AI to not do. Those are the people whose culture Anthropic descends from, and by their measure, those exclusions make this the least evil path.

The day self hosted models catch up with Anthropic’s capabilities is when they will fully lose their shit. This day can’t come soon enough

They do, and they are still actively hiring.

https://job-boards.greenhouse.io/anthropic/jobs/5066977008 https://job-boards.greenhouse.io/anthropic/jobs/5239733008

Again, HN fell for the marketing and believed everything they did was for "safety".