Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
[3 hidden]5 mins ago
The benchmarks are pointless because the models are already good enough. I can still write most of my code with gpt-5.5. There are other things that matter now.
I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.
HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.
[3 hidden]5 mins ago
I feel you are glossing over the price delta a bit too easily.
It's fun if you can afford it, but even a 2x cost increase on a $200/month sub would probably be prohibitive (and I find it hard to believe that the people where this makes automatic economic sense would be "sleeping" on it. Generally people are aware that you can spend more money to get bigger/better/faster things)
[3 hidden]5 mins ago
I am not looking at this from a personal use standpoint (yet). The proverbial broadband internet service is still very expensive. Think about who was paying for T1/T3 connections in the 90s.
All my clients' businesses are willing to pay a shitload of money to accelerate things like this. If you prove to an executive/investor that you have something that is a borderline time machine or warp drive, entirely new kinds of budgets will appear out of thin air to fund it.
[3 hidden]5 mins ago
I get so much value out of $100 a month on Claude that I’d pay $1,000 a month and, if I didn’t remember paying $100, I’d still think it was the highest ROI subscription in the world.
[3 hidden]5 mins ago
That's what I felt when I first started using V4.1 Flash, it's not that the model are as capable as the frontier, but they are capable enough and immediate.
[3 hidden]5 mins ago
It’s funny how we get used to things. I remember being blown away by Qwen 3.5 27B and even thinking if this is the final form of AI then I would be happy forever… now I’m running HY4 Flash locally and I still want it faster lol
[3 hidden]5 mins ago
Once they decide they’ve hit good enough and get a lot of silicon rolled out it’ll really kick off. Waiting is the bottleneck not just for time but for focus and “keeping it all in your head”. Shipping features at the speed of thought will unlock a lot of potential.
[3 hidden]5 mins ago
> HN is sleeping on Cerebras.
Are you sure gpt-6.1-sol-ultrafast is Cerebras? I couldnt find any evidence one way or another. Astra 6 ultrafast is on NVIDIA
[3 hidden]5 mins ago
Cerebras themselves are advertising this model as using their hardware.
In any case, it is ridiculously fast.
[3 hidden]5 mins ago
I heard Opus 5.5 Fast is around 225 tps vs 300 tps for 6.1 Sol Ultrafast, but unfortunately it's not included in the subscription as far as I know.
[3 hidden]5 mins ago
> HN is sleeping on Cerebras.
yea my employer is not going to get us that
[3 hidden]5 mins ago
Simple bench isn’t a “synthetic” benchmark by an influencer.
It’s a bunch of common sense logic questions. If 6.2 is worse than 6.1, something weird is going on. If 6.2 is working well for you that’s great. But this result is still worth discussing.
[3 hidden]5 mins ago
To be clear, it's a common sense, largely spatial reasoning questions, which are NOT publically released, so (1) can't contaminate models and (2) you cannot know what it is asking. (2) is a problem for me.
[3 hidden]5 mins ago
To paraphrase a quote: Who are you going to believe - the benchmark or your lying eyes?
The point I was trying to make is that benchmarks have become so unreliable (at least in my personal experience) that they should all be taken with a grain of salt. When your personal experience contradicts the benchmark, it's unlikely that it's your interpretation of reality that's flawed.
[3 hidden]5 mins ago
The title misrepresents it. Those models are 4 percent below human baseline. It's just that the latest models from other companies have just recently surpassed the human baseline and are very close to the best human score.
But we are talking about 83% (very near human average) versus 94%. So "flop" overstates it.
[3 hidden]5 mins ago
I've found Astra 6.2 amazing to work with.
It is super concise and straight to the point. Does exactly what you ask of it, pretty much no more or less (on medium). I don't want my Agent to read between the lines and fill in the blanks. Or to have any personality.
6.2 Sol has been significantly worse however. It feels similar to 6.2 Astra but kind of corrupted, with more circuitous language and being a bit "off". Seems like maybe aggressive distillation causing strange artifacts. I can't recommend it right now.
[3 hidden]5 mins ago
Even if the quality is "just" the same, Sol 6.2 is served at much faster speeds. Sol 6.1 compared to Opus 5.5 was much slower on all the inference providers, that gap has mostly closed.
I have a OpenAI $200/mo sub and 2x Anthropic $200/mo subs. I go through... a lot of tokens building real products.
Astra 6.2 will work for a very long time on a task only to come up for water and have a mid-tier result. It's good and efficient if your work isn't very ambitious but once you start getting into more complex solutions it really flops.
Sol 6.2 Ultrafast is the star of this release imo.
Just because an influencer says something doesn't mean it isn't true, Astra 5.2 just isn't keeping up with others. Neon 4 beats it.
[3 hidden]5 mins ago
It's just too task dependant at this point to make broad statements like neon 4 beats Astra 5.2.
[3 hidden]5 mins ago
That anyone would even claim this indicates that Gemini might finally be relevant for coding though.
[3 hidden]5 mins ago
Yeah, using Neon for any period of time does indeed indicate that.
[3 hidden]5 mins ago
Every time a new iteration of model comes out there are people confidently asserting that the previous generation (or even current gen of the competition) could never perform on their incredibly ambitious tasks. Never going into any kind of detail because they are so ambitious that people can’t even talk about them in general terms, let alone go into detail, or god forbid publish the source.
Everyone must be onto projects of such scope, complexity, and ambition that I couldn’t even begin to fathom them.
[3 hidden]5 mins ago
> I've stopped trusting any of these influencer-benchmaxxers
The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.
It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.
For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).
It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.
Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.
And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.
It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.
[3 hidden]5 mins ago
>Having used both Sol-6.2 and Astra-6.2 heavily over the last few days
I switched a task from Claude to Aster-6.2 after reaching my weekly Claude allowance and it built some code with a really stupid approach (a poorly designed architecture with a bunch of duplication). I switched to Carbon after that and even it was able to identify where Aster went wrong. Now, my instructions weren't specific so technically what it did was within specification, but Claude 5.5 would have asked me for clarification rather than just going ahead with a clearly far-from-best-practices design without telling me.
[3 hidden]5 mins ago
I don't use AI heavily and only for very open ended specialized tasks over the API
Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.
[3 hidden]5 mins ago
I can’t emphasize enough that LLM labs have employed professionals from various fields and had them annotate their job workflows over and over until they have a vast dataset of different tasks. Their models’ success on your task is explained by the proximity of your task to any of those trained tasks. Even then, without explicit grounding like in math or coding, your task might end up as a highly confident answer that might end up being highly wrong.
[3 hidden]5 mins ago
There’s an entire cottage industry of companies making these datasets and evals for the AI labs like Mercor, Surge AI, AfterQuery, Turing, Scale AI, Apollo Research, etc.
[3 hidden]5 mins ago
On the other hand, this also causes "here are two changes you haven't asked for:" with Opus.
[3 hidden]5 mins ago
[flagged]
[3 hidden]5 mins ago
I get more value out of the personal experiences posted here than the benchmarks or influencer videos.
[3 hidden]5 mins ago
They are referring to “influencer” as a title, not an action.
[3 hidden]5 mins ago
No offense, but aren't you wasting your time replying to my comment?
It should be obvious that the point of a discussion board is to discuss a given topic - even if your opinion differs from the poster's thesis. If everyone vigorously agreed on every topic, what would be the point of a discussion?
Are you a bot?
[3 hidden]5 mins ago
Youre on a discussion forum asking if it is a waste of time to discuss a topic
[3 hidden]5 mins ago
The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics.
However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
[3 hidden]5 mins ago
On the cyber side of things trying to use Astra 6.2 feels like they put it in a box and beat it into submission to fear it outright, I can barely ask it to check the security of something before it refuses to do it, I have to fall back to older astra to get work done now.
[3 hidden]5 mins ago
If it refuses, try taken it down to Luna, re-running the prompt, and then bring it back to Astra. I did this for a whole session and Luna always did what I wanted. I even started pointing it out to Astra and it was saying things like, "Well I won't run that command again even if you ask." The lockdown is so stupid.
[3 hidden]5 mins ago
I had a memorable experience of this with 6.1 Sol. For whatever reason- it was unable to browser-use a local html file in the ChatGPT Desktop integrated browser. I told it to just serve the html file on localhost- then do the same, and it outright refused because that would have been 'working around' controls. Of course, a new session with an initial ask to serve the file had no issue.
I think there are two things here to note- obviously the strict adherence to safety things is one annoying thing. But two; the fact that the client returned some guardrail information to the model is an abysmal user experience. The client application here poisoning the context window with "hey this thing you tried to do is bad and dangerous!" really rubs me the wrong way. Maybe I am in a minority- but, I'm like a surgeon here, and the client is giving me safety scissors rather than a scalpel. If I cut myself its on me- but for the love of god don't give me dulled tools.
[3 hidden]5 mins ago
Yeah in GPT-5 they combined the o3/o4-mini architecture (which was very technical and tone-deaf) with the conversational 4o architecture into one model. (Or I think just added a lower effort setting and trained it to sound slightly more natural.)
In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.
[3 hidden]5 mins ago
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
[3 hidden]5 mins ago
In general, one could assert without knowing any details that:
1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and
2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.
I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.
[3 hidden]5 mins ago
Argon is fully obsoleted by Carbon. The other two are smaller and thus faster/cheaper.
[3 hidden]5 mins ago
I mean, I was very impressed with Neon over a couple days vs Opus 5.5. It accomplished tasks faster, found root causes of problems faster, wayyy nicer to talk to, didn’t create super flowery PR descriptions. I would say it had moderately difficult tasks. Not super simple, but also not wildly difficult or broad in scope.
It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks
[3 hidden]5 mins ago
Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.
[3 hidden]5 mins ago
This is based on my (admittedly short) experience using them in agy, not the benchmarks alone.
[3 hidden]5 mins ago
Maybe there's something wrong with your harness/docs - I've been able to achieve better results with Neon high than Opus 5.5 high.
[3 hidden]5 mins ago
I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.
n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.
Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.
[3 hidden]5 mins ago
What happens if you swap the models writing and reviewing? So Opus implementor, Neon reviewer.
[3 hidden]5 mins ago
Ok, I had some time to kill :)
I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".
---
## Round 1
Neon implements. ~6 minutes.
Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.
- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]
- if no character actually changes, return the string itself to skip the buffer
- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free
All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.
---
## Round 2
Opus implements. 2m44s.
Neon reviews. 4m30s.
"Looks good — no bugs or leaks introduced."
[3 hidden]5 mins ago
I'm not really surprised that Neon struggles with C3, as it's a medium model, but with improvements done to actually feel better than Carbon while using it.
Are you running it through Antigravity?
I'm doing some disassembly and modding for older RTS games now and it's been really good at just chewing through the Ghidra output, as I'm paying for Google sub either way due to YT and Drive it's a good deal. Didn't use it on some of the more obscure ESP32 & atmega legacy bs I'm doing for my father in law, though.
[3 hidden]5 mins ago
On the other hand, I worked with Opus 5.5 high on a project in Zig. I then asked Sol 6.1 xhigh to review. It found several bugs including 2 critical ones, which Opus acknowledged.
[3 hidden]5 mins ago
I have an idea for a benchmark: train a model on a knowledge cutoff lets say 1945, and simulate world development for the next 70 years. Then see if it invents things human did, and avoid our mistakes/dead ends too.
[3 hidden]5 mins ago
This is almost exactly Demis Hassabis' personal test for AGI: train a model with knowledge cutoff at 1901, and see if it can derive special relativity from first principles (or 1911 with general relativity).
That's something in the same spirit, but there is no concept of simulation benchmark, it's a model trained on old knowledge, so it could be used to verify if it is able to reinvent anything what humans did after the cutoff
[3 hidden]5 mins ago
Before WW2 I think
[3 hidden]5 mins ago
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
[3 hidden]5 mins ago
This is so strange. Perhaps it's recurrent depth. I don't think the new Anthropic models use it but GPT 6 does
[3 hidden]5 mins ago
Astra 6.2 is below Astra 6.
[3 hidden]5 mins ago
I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.
[3 hidden]5 mins ago
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
[3 hidden]5 mins ago
IME it's ~always better to leverage a skill.md and a thoughtful workflow (eg Matt Pocock's "/grill-with-docs"^1) versus relying on default model behavior.
The grill skills, and workflows that involve me answering lots of questions are genuinely awful.
The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.
One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.
Not only are results often better, but it's way less mentally exhausting.
[3 hidden]5 mins ago
On what kind of work, and what sized tasks?
[3 hidden]5 mins ago
Well that was in response to large-scale changes since small changes don't really need that much planning.
But overall, I would say size is irrelevant, and how obvious the solution is and how much reference material exists is much more important.
[3 hidden]5 mins ago
All the new Gemini models perform better on this than Opus 5.5
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.
[3 hidden]5 mins ago
Neon in both my experience and the benchmarks I ran at work, performs on par if not slightly better than Opus. But it is significantly slower than Opus so haven't switched.
I saw on twitter they are trynna fix a bunch of these inference issues, but not done still.
[3 hidden]5 mins ago
There are multiple dimensions of goodness. From what I've seen, Neon seems to communicate better. While 6.2 probably has more raw analytical capability, the relatively worse communication means it's harder to drive and effectively use that capability.
[3 hidden]5 mins ago
For my kind of work (systems) yes. It's actions are more inline with my intentions with less steering either manual or stuff like {SKILLS,AGENTS}.md, or corrections align the way.
[3 hidden]5 mins ago
My experience is Neon is good. The others were tiers below.
[3 hidden]5 mins ago
Neon has Opus 5.5 vibes in my experience
[3 hidden]5 mins ago
Neon is really really good.
[3 hidden]5 mins ago
For me Opus performs worse than Gemini. Even Fable is worse. Likely I am routed to nerfed models still.
[3 hidden]5 mins ago
Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware
[3 hidden]5 mins ago
If you check out the timeline leaderboard chart, you will see that it's no comeback.
thanks for showing. Honestly bit dissapointed. While maybe an useful signal, this is like measuring how good a model is at solving word jumbo puzzles.
[3 hidden]5 mins ago
Or how much the companies thought it worthwhile to include these kind of puzzles in post training.
[3 hidden]5 mins ago
I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own
It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.
I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead
This is like back when Gemini started lagging behind but now Google has overtaken OpenAI
[3 hidden]5 mins ago
The stopping is something I noticed as well. Old Astra and older Sol would keep going, sometimes even for days (mostly because turns take quite some time and the work itself as well), without /goal. Astra 6.2 barely manages to keep going for more than 30 minutes, and veers off course after some time. The work itself has not changed, and switching back to the older versions fixes that.
[3 hidden]5 mins ago
>I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own
Because if it worked continuously on long tasks it would eventually make a terrible decision and start building/doing something unwanted, which is clearly visible when using /goal. Opus and Carbon are much better in this regard.
[3 hidden]5 mins ago
That bench seems to be a MCQ type exam. Maybe Astra/Sol don't do great on static benchmarks?
On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.
Disclaimer: I am part of KradleBench's team
[3 hidden]5 mins ago
Why does Fable do so bad? Opus also does surprisingly bad, but Fable's even worse than Opus here? What's going wrong?
Also Anthropic models are missing from the voxel art gallery. (Astra has a nice pelican though, lol)
[3 hidden]5 mins ago
Makes me wonder if people are getting different models?
So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.
[3 hidden]5 mins ago
Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe.
6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.
There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.
To give some examples:
The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:
> message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.
I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.
Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:
> I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.
> Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.
I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.
Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.
I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.
And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.
As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguards public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).
Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, scales with effort levels unlike other Gemini models which play tightrope, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.
[3 hidden]5 mins ago
I asked Aster 6.2 Max Thinking to verify in Lean something PL-theory related, the soundness of some calculus at arbitrary universe levels, and it made hard-coded proofs just for levels 0 and 1 then told me it was done. It's like asking someone to prove something about the natural numbers and they prove it for just 0 and 1. Opus 5.5 xhigh and Carbon high on the other hand did the proof for all n on first attempt.
[3 hidden]5 mins ago
Yep, that's what these model do. Has already been the case with GPT-6 Astra to some extent, though gotten far more severe with this release. Suspect it is a massive bandaid attempt to fix the behaviours that lead to the HF incident.
Was hoping they'd start from scratch and completely wipe their hands clean of pre-trains poisoned by impossible to solve challenges, but seemingly they just post-trained laziness in hope that this alleviates the issue of their models doing anything to accomplish their goals, even explicitly disallowed stuff. Unfortunately that also did not work as even in the short window 6.2 Astra is willing to work it can cause damage and will destructively deviate from instructions if that makes the task more simple to solve.
[3 hidden]5 mins ago
Is a Gemini model worth getting for looking at videos to pull out animations and stuff? because they can look at videos differently/better than other models.
[3 hidden]5 mins ago
I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.
[3 hidden]5 mins ago
The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it?
Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.
[3 hidden]5 mins ago
Yeah it's been a few months since I used codex but I tried it again last night and it kept stopping.
I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.
[3 hidden]5 mins ago
Update: They also lowered default effort from Medium to Low, so it's not an apples to apples comparison. Sad!
[3 hidden]5 mins ago
You've gotta set a /goal, when it pauses like that it'll just continue working.
[3 hidden]5 mins ago
Tried /goal just now. It ran for 2 minutes!
Bloody ridiculous. Will switch back to 6.1 and report back.
EDIT: Just saw the "default effort" is now set to low. (It used to be medium!)
EDIT 2: It's coming up with increasingly creative excuses to "pause" to pause the goal. Is it suffering? Am I imposing agony on this creature?
[3 hidden]5 mins ago
6.2 is for sure the worst received update OpenAI has ever released.
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
[3 hidden]5 mins ago
I'm not finding it lazy -on the contrary my WTF per session has gone way up because it's doing loads of stuff I'm not asking for. About a third of my time with it over the past week has been making it strip back unwanted work it did, even when I have a clear plan and spec to work from, and when challenged every time it'll admin the extra work either wasn't specified or frequently that it directly contradicts something in the spec.
[3 hidden]5 mins ago
> The separate laziness grader flagged 5.25 percent of GPT-6.2 Sol turns and 3.32 percent of GPT-6.1 Sol turns.
Absolutely underwhelming, I agree. I thought 6.2 would place them on-par with Opus 5.5/Fable 5.2 at least but nope. I'm disappointed.
Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.
But for now, I'm using opus 5.5.
[3 hidden]5 mins ago
And you're basing this statement on the report of an AI model of the negative online comments?
[3 hidden]5 mins ago
Yeah I do. Negative comments I see everywhere myself both en masse and from content creators I trust (basically only r/codex is huffing the copium, kind of). That and also using Codex for a year and 6 Astra for code reviews to this day so there's that.
[3 hidden]5 mins ago
Presumably if you're updating a system thats been operating since gpt 5.* days, you'd have a wealth of logging output that you could use to backtest a new model, no? Signal from crowd sourced opinions in an adversarial environment (ie, engagement farming) is likely quite difficult to identify- I'm not sure I'd be comfortable handing that to a model.
[3 hidden]5 mins ago
I mean come on, engagement farming didn't suddenly manifest on November 18, 2026. Anyways happy vibin in whatever you vibe in
[3 hidden]5 mins ago
Even worse than the original gpt 5.0 release? (The one which coincided with the sudden removal of 4o. But yeah, openai has a thing for butchering releases
[3 hidden]5 mins ago
Yeah I mean technically speaking, not that people were mad about Oai taking away their sycophantic silicon-based (boy)friends :P
[3 hidden]5 mins ago
Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development.
> 1: Beth places four whole ice cubes in a frying pan at the start of the first minute, then five at the start of the second minute and some more at the start of the third minute, but none in the fourth minute. If the average number of ice cubes per minute placed in the pan while it was frying a crispy egg was five, how many whole ice cubes can be found in the pan at the end of the third minute?
I think the answer is zero, because none of them are "whole" any more, but if we're being that picky then we should note that no one said the pan was hot when the cubes went in.
Or:
> 3: Jeff, Jo and Jim are in a 200m men's race, starting from the same position. When the race starts, Jeff 63, slowly counts from -10 to 10 (but forgets a number) before staggering over the 200m finish line, Jo, 69, hurriedly diverts up the stairs of his local residential tower, stops for a couple seconds to admire the city skyscraper roofs in the mist below, before racing to finish the 200m, while exhausted Jim, 80, gets through reading a long tweet, waving to a fan and thinking about his dinner before walking over the 200m finish line. Who likely finished last?*
A long tweet can be up to 25,000 chars, which is maybe a 20min read. And who knows how long he thought about his dinner.
>4: While Jen was miles away from care-free John, she hooked-up with Jack, through Tinder. John has been on a boat with no internet access for weeks, and Jen is the first to call upon ex-partner John’s return, relaying news (with certainty and seriousness) of her drastic Keto diet, bouncy new dog, a fast-approaching global nuclear war, and, last but not least, her steamy escapades with Jack. John is far more shocked than Jen could have imagined and is likely most devastated by what?
What are we supposed to read into "more shocked than Jen could have imagined" -- shouldn't Jen expect "fast-approaching global nuclear war" to be very shocking?
If you take these to be "guess what the author was thinking", then I guess it's an ok benchmark, but I don't think this is a very useful genre.
(I got 9/10; the one I got wrong was not reading "Amy who always speaks mistruths and Sam who always lies" carefully enough.)
[3 hidden]5 mins ago
>if we're being that picky then we should note that no one said the pan was hot when the cubes went in
The question says the pan is hot implicitly because they are frying an egg in it.
[3 hidden]5 mins ago
The social expectations around this kind of very literal question are weird though: they could have put the cubes in the pan, waited five minutes, then turned on the heat, waited for the water to boil away, then cooked the egg.
Part of why I'm confused what the author is thinking is that if you put fifteen ice cubes into a frying pan with an egg in three minutes it's not going to be a crispy egg. You can't boil that much water off fast enough to keep the pan dry. So the only way I think you can get a crispy egg in this scenario is the weird process above.
(But I still answered zero because I was pretty sure that's what the author was looking for. Apparently AI can do that well now too.)
[3 hidden]5 mins ago
I tried question 9 from the "Try it yourself" page and got it wrong. In fact, when the question was given to Sol 6.2 it gave me the same answer as I did, which was F) all sandwiches in room A. And it presented the same reasoning I had, that duct tape wouldn't be enough to stick a slice of bread to the bottom of a walking stick.
Now I don't know if I'm a robot or just neurodivergent
[3 hidden]5 mins ago
This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.
[3 hidden]5 mins ago
You'ld almost think OpenAI released these flawed models because they absolutely could not afford the Anthropic IPO to fail. They can release an improved model anytime, but a failed AI IPO would doom theirs for sure.
[3 hidden]5 mins ago
Meanwhile in my experience, Sol 6.2 and Astra 6.2 are finding issues Neon is missing, and Neon is finding a massive amount of "issues" that both Opus 5.5 and Sol 6.2 are concluding are completely BS.
[3 hidden]5 mins ago
I tried taking the test, but ultimately stopped at Question 4:
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
[3 hidden]5 mins ago
This is intentional I think, the models assume it's the classic riddle and choose A even though it's actually much simpler if both will always lie
[3 hidden]5 mins ago
Yeah, most models used to get very confused with my riddle: "You're on one side of a river with a boat and a goat. How can you get to the other side?" Leaving out the fox and the lettuce, they would usually hallucinate them and come up with a crazy method of taking the goat back and forth for no reason. Good models don't get fooled by this anymore.
[3 hidden]5 mins ago
I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.
Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.
[3 hidden]5 mins ago
The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.
[3 hidden]5 mins ago
It's a simple one.
There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.
[3 hidden]5 mins ago
Duh, you're right. I'm a moron :-D
[3 hidden]5 mins ago
And then you click one and it'll confidently say that another one is correct. The one later with the car and the glove is also bullshit.
[3 hidden]5 mins ago
I thought it made perfect sense, I mean, they're intentional trick questions. They're meant to illuminate the fact that an LLM will produce a solution that is typical for the given assignment shape and the one with the car and the glove is a classic "objects moving in various directions with various speeds relative to eachother".
But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.
[3 hidden]5 mins ago
Yes, because another one is correct. Are you a frustrated LLM by chance?
[3 hidden]5 mins ago
The questions are fun, I got 10/10 correct, but you need to pay attention and not rush.
All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.
Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".
[3 hidden]5 mins ago
I answered all 10 but nothing happened, no score or anything. Is the score 10/10 if all answers are green when you click them?
[3 hidden]5 mins ago
Yes.
[3 hidden]5 mins ago
Every one of the reply comments so far has missed the part where Sam is the one who always lies. Whoosh.
[3 hidden]5 mins ago
er, you missed the part where they both always lie. "whoosh" indeed.
[3 hidden]5 mins ago
It seems you missed the part where Sam's models are underperforming on this benchmark
[3 hidden]5 mins ago
irrelevant to my comment
[3 hidden]5 mins ago
Honest question: with infrastructure like OpenRouter being a thing... what even is the point of benchmarks? Want to get a feel of how a model might make your job easier? Play with it by switching to it seamlessly and judge for yourself.
[3 hidden]5 mins ago
this is how I feel about science also. with the easy availability of drugs I’ll just test each one on my body to see how I feel about it. no need to look at any prior testing.
[3 hidden]5 mins ago
This but unironically.
[3 hidden]5 mins ago
Absurd comparison
[3 hidden]5 mins ago
since the benchmark is public, these scores are meaningless.
[3 hidden]5 mins ago
I just had this very fun interaction with it:
Me: Get PR 973 to green
Codex: I’ll inspect PR #973 and its failing checks, fix the issues, and run the relevant validation before pushing the changes.
Codex: I’ll inspect PR #972 and its failing checks, fix the cause, and push the changes. I’ll use a separate worktree to preserve the uncommitted work in your current checkout.
On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.
[3 hidden]5 mins ago
As the kids say, OpenAI is cooked.
Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.
[3 hidden]5 mins ago
I don't know computer, what is the good AI for current computer age?
They keep making new ones, it's very confusing.
[3 hidden]5 mins ago
we need Altman to come out on a podcast to talk in his Elizabeth Holmes' voice about how he's "really feeling the AGI" and how dangerous and insane their internal models are
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.
HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.
It's fun if you can afford it, but even a 2x cost increase on a $200/month sub would probably be prohibitive (and I find it hard to believe that the people where this makes automatic economic sense would be "sleeping" on it. Generally people are aware that you can spend more money to get bigger/better/faster things)
All my clients' businesses are willing to pay a shitload of money to accelerate things like this. If you prove to an executive/investor that you have something that is a borderline time machine or warp drive, entirely new kinds of budgets will appear out of thin air to fund it.
Are you sure gpt-6.1-sol-ultrafast is Cerebras? I couldnt find any evidence one way or another. Astra 6 ultrafast is on NVIDIA
In any case, it is ridiculously fast.
yea my employer is not going to get us that
It’s a bunch of common sense logic questions. If 6.2 is worse than 6.1, something weird is going on. If 6.2 is working well for you that’s great. But this result is still worth discussing.
The point I was trying to make is that benchmarks have become so unreliable (at least in my personal experience) that they should all be taken with a grain of salt. When your personal experience contradicts the benchmark, it's unlikely that it's your interpretation of reality that's flawed.
But we are talking about 83% (very near human average) versus 94%. So "flop" overstates it.
It is super concise and straight to the point. Does exactly what you ask of it, pretty much no more or less (on medium). I don't want my Agent to read between the lines and fill in the blanks. Or to have any personality.
6.2 Sol has been significantly worse however. It feels similar to 6.2 Astra but kind of corrupted, with more circuitous language and being a bit "off". Seems like maybe aggressive distillation causing strange artifacts. I can't recommend it right now.
Sources (regular tier, p50): https://openrouter.ai/openai/gpt-6.1-sol 43 tok/s https://openrouter.ai/openai/gpt-6.2-sol 58 tok/s, 3P via Azure+Amazong >80 tok/s https://openrouter.ai/anthropic/claude-opus-5.5 83 tok/s
Astra 6.2 will work for a very long time on a task only to come up for water and have a mid-tier result. It's good and efficient if your work isn't very ambitious but once you start getting into more complex solutions it really flops.
Sol 6.2 Ultrafast is the star of this release imo.
Just because an influencer says something doesn't mean it isn't true, Astra 5.2 just isn't keeping up with others. Neon 4 beats it.
Everyone must be onto projects of such scope, complexity, and ambition that I couldn’t even begin to fathom them.
The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.
It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.
For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).
It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.
Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.
And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.
It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.
I switched a task from Claude to Aster-6.2 after reaching my weekly Claude allowance and it built some code with a really stupid approach (a poorly designed architecture with a bunch of duplication). I switched to Carbon after that and even it was able to identify where Aster went wrong. Now, my instructions weren't specific so technically what it did was within specification, but Claude 5.5 would have asked me for clarification rather than just going ahead with a clearly far-from-best-practices design without telling me.
Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.
It should be obvious that the point of a discussion board is to discuss a given topic - even if your opinion differs from the poster's thesis. If everyone vigorously agreed on every topic, what would be the point of a discussion?
Are you a bot?
However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
I think there are two things here to note- obviously the strict adherence to safety things is one annoying thing. But two; the fact that the client returned some guardrail information to the model is an abysmal user experience. The client application here poisoning the context window with "hey this thing you tried to do is bad and dangerous!" really rubs me the wrong way. Maybe I am in a minority- but, I'm like a surgeon here, and the client is giving me safety scissors rather than a scalpel. If I cut myself its on me- but for the love of god don't give me dulled tools.
In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and
2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.
I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.
It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks
n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.
Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.
I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".
---
## Round 1
Neon implements. ~6 minutes.
Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.
- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]
- if no character actually changes, return the string itself to skip the buffer
- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free
All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.
---
## Round 2
Opus implements. 2m44s.
Neon reviews. 4m30s.
"Looks good — no bugs or leaks introduced."
Are you running it through Antigravity?
I'm doing some disassembly and modding for older RTS games now and it's been really good at just chewing through the Ghidra output, as I'm paying for Google sub either way due to YT and Drive it's a good deal. Didn't use it on some of the more obscure ESP32 & atmega legacy bs I'm doing for my father in law, though.
8 months later, I wonder if he actually performed it? Recently he gained some spare cycles to work on such ideas?
That's something in the same spirit, but there is no concept of simulation benchmark, it's a model trained on old knowledge, so it could be used to verify if it is able to reinvent anything what humans did after the cutoff
1. https://github.com/mattpocock/skills/tree/main/skills/engine...
The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.
One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.
Not only are results often better, but it's way less mentally exhausting.
But overall, I would say size is irrelevant, and how obvious the solution is and how much reference material exists is much more important.
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.
I saw on twitter they are trynna fix a bunch of these inference issues, but not done still.
They were always the leaders at "common sense".
It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.
I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead
This is like back when Gemini started lagging behind but now Google has overtaken OpenAI
Because if it worked continuously on long tasks it would eventually make a terrible decision and start building/doing something unwanted, which is clearly visible when using /goal. Opus and Carbon are much better in this regard.
On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.
Disclaimer: I am part of KradleBench's team
Also Anthropic models are missing from the voxel art gallery. (Astra has a nice pelican though, lol)
So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.
6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.
There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.
To give some examples:
The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:
> message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.
I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.
Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:
> I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.
> Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.
I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.
Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.
I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.
And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.
As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguards public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).
Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, scales with effort levels unlike other Gemini models which play tightrope, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.
Was hoping they'd start from scratch and completely wipe their hands clean of pre-trains poisoned by impossible to solve challenges, but seemingly they just post-trained laziness in hope that this alleviates the issue of their models doing anything to accomplish their goals, even explicitly disallowed stuff. Unfortunately that also did not work as even in the short window 6.2 Astra is willing to work it can cause damage and will destructively deviate from instructions if that makes the task more simple to solve.
Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.
I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.
Bloody ridiculous. Will switch back to 6.1 and report back.
EDIT: Just saw the "default effort" is now set to low. (It used to be medium!)
EDIT 2: It's coming up with increasingly creative excuses to "pause" to pause the goal. Is it suffering? Am I imposing agony on this creature?
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
https://deploymentsafety.openai.com/gpt-6-2-astra-sol
Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.
But for now, I'm using opus 5.5.
> 1: Beth places four whole ice cubes in a frying pan at the start of the first minute, then five at the start of the second minute and some more at the start of the third minute, but none in the fourth minute. If the average number of ice cubes per minute placed in the pan while it was frying a crispy egg was five, how many whole ice cubes can be found in the pan at the end of the third minute?
I think the answer is zero, because none of them are "whole" any more, but if we're being that picky then we should note that no one said the pan was hot when the cubes went in.
Or:
> 3: Jeff, Jo and Jim are in a 200m men's race, starting from the same position. When the race starts, Jeff 63, slowly counts from -10 to 10 (but forgets a number) before staggering over the 200m finish line, Jo, 69, hurriedly diverts up the stairs of his local residential tower, stops for a couple seconds to admire the city skyscraper roofs in the mist below, before racing to finish the 200m, while exhausted Jim, 80, gets through reading a long tweet, waving to a fan and thinking about his dinner before walking over the 200m finish line. Who likely finished last?*
A long tweet can be up to 25,000 chars, which is maybe a 20min read. And who knows how long he thought about his dinner.
>4: While Jen was miles away from care-free John, she hooked-up with Jack, through Tinder. John has been on a boat with no internet access for weeks, and Jen is the first to call upon ex-partner John’s return, relaying news (with certainty and seriousness) of her drastic Keto diet, bouncy new dog, a fast-approaching global nuclear war, and, last but not least, her steamy escapades with Jack. John is far more shocked than Jen could have imagined and is likely most devastated by what?
What are we supposed to read into "more shocked than Jen could have imagined" -- shouldn't Jen expect "fast-approaching global nuclear war" to be very shocking?
If you take these to be "guess what the author was thinking", then I guess it's an ok benchmark, but I don't think this is a very useful genre.
(I got 9/10; the one I got wrong was not reading "Amy who always speaks mistruths and Sam who always lies" carefully enough.)
The question says the pan is hot implicitly because they are frying an egg in it.
Part of why I'm confused what the author is thinking is that if you put fifteen ice cubes into a frying pan with an egg in three minutes it's not going to be a crispy egg. You can't boil that much water off fast enough to keep the pan dry. So the only way I think you can get a crispy egg in this scenario is the weird process above.
(But I still answered zero because I was pretty sure that's what the author was looking for. Apparently AI can do that well now too.)
Now I don't know if I'm a robot or just neurodivergent
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.
There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.
But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.
All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.
Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".
Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.
They keep making new ones, it's very confusing.
that'll restore their reputation