Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
eptcyka [3 hidden]5 mins ago
Don’t apologize for x, just don’t post x links.
encypruon [3 hidden]5 mins ago
I think it's the best possible approach. It lets you refer to content that's unfortunately stuck on the site, while still signaling clearly that it's something to avoid if possible and to apologize for if not. Omitting the link just removes information without influencing consensus on whether to use X.
edit: I wish it would let me view the darn screenshot tho :c
uoaei [3 hidden]5 mins ago
There are plenty of alternatives to link to X content without hitting X's servers.
flaburgan [3 hidden]5 mins ago
Do you mind listing some? I only know xxcancel.com
throwaway219450 [3 hidden]5 mins ago
I removed it, but it's frustrating when the options are someone's (reasonable) post on X, or slop-spam aggregators. I don't really want to drive visits to either, but at least the X poster can get the views. I don't really like xcancel either as it still encourages indirect engagement with X.
CharlieDigital [3 hidden]5 mins ago
I use the frontier models only in plan mode; coding is held out to cheaper models.
I run Paseo as an orchestrator with agent profiles. "Distinguished Member of Technical Staff" -> Fable, plan mode. "End-User Customer Advocate" -> Astra, plan mode. A few others. Basically, these agents (with profile prompts) can only be "consulted" for input on design elements; it's OK if they produce a bit of verbosity here. This has worked fantastically for me, really awesome to see a self-constructed team own the work end-to-end.
A "Director of Engineering" (6.1 Sol or Opus 5.5) also in plan mode organizes the team for the task and then owns coordination and instruction. You can think of this in essence like a "model router".
Coding is handled by Haiku 5.5/GPT 6 Luna/DeepSeek v4.1 Flash. One "Principal Engineer" Sonnet 5.5 or 6.1 Sol for some tasks.
grim_io [3 hidden]5 mins ago
You are deep in it.
You may be larping for no additional benefit vs just using Opus 5.5 and just asking it to implement a feature.
CharlieDigital [3 hidden]5 mins ago
Nah; I can use my token budget across both providers this way, use tokens more sparingly with smaller models for heavy text in/out, and you'd be surprised when the agents talk to each other and "role play" how much this improves things like the UX as well as technical design. Distinguished Member of Technical Staff, for example, is asked to "expand the solution space" which I find has a strong effect on approaches considered.
This also helps with context growth since the sub-agents do the heavy lifting (e.g. search, read docs) and just provide the key bits back to the lead.
Is it "larping"? I ran an eval today on a contract ingest and extract agent I'm working on. Changing the <role> fragment of the prompt lifted the results by +7 points by simply changing the text to the effect of "You are a senior partner at a distinguished legal firm that focuses on reading licensing contracts..." and some bits about the content the agent is reviewing + objective.
My conclusion: the agents like to LARP.
ttul [3 hidden]5 mins ago
I've noticed the line breaks issue. It's easily fixed with a formatter like prettier and, frankly, that kind of quality-of-life thing is best handled deterministically anyway.
rsfern [3 hidden]5 mins ago
Doesn’t this mess up the context though? I use ruff and my agents get confused if I edit and run ruff out of band. How do people deal with that? I’m always having to remind agents to re-read files because they’ve changed
IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
skrtskrt [3 hidden]5 mins ago
They also seem to really bias towards tacking on new code to "work around" what's there, rather than breaking down what's there into smaller more composable bits that can serve the pre-existing use case and the new requirement elegantly.
hedgehog [3 hidden]5 mins ago
Is there a reasonable way to run a representative subset myself?
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.
furyofantares [3 hidden]5 mins ago
For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.
WASDx [3 hidden]5 mins ago
Great benchmark!
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
giancarlostoro [3 hidden]5 mins ago
GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.
AnotherGoodName [3 hidden]5 mins ago
You unknowingly substituted the name of another model when you’re otherwise talking about the competitor fwiw. Some if those in the reply chain didn’t even seem to notice.
That’s a little telling of the state of ai atm though isn’t it? It’s all pretty similar. The latest Opus/GPT/Gemini/Deepseek/Kimi on medium does indeed do the job
spijdar [3 hidden]5 mins ago
Why not GPT-{5.6, 6, 6.1} Sol? They're all cheaper than GPT 5.5, and as "smart", if not "smarter", at least to me.
Though, it has been a while since I've used 5.5.
ttul [3 hidden]5 mins ago
Surely they mean Opus-5.5...
singingtoday [3 hidden]5 mins ago
I have no data to support this, but when working on SaaS systems I find anthropic models more effective.
But for webdev, deep research, code review, and a few other tasks I find the current 6.1 sol better.
But I use both. And I use codex more.
foota [3 hidden]5 mins ago
Wouldn't it be better to enforce pass via the harness and then measure cost/quality since that's more what people are likely to do?
mordae [3 hidden]5 mins ago
This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?
meffmadd [3 hidden]5 mins ago
Are the raw outputs published somewhere?
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
drivebyhooting [3 hidden]5 mins ago
First of all, what happened to Max and Ultra?
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
4b11b4 [3 hidden]5 mins ago
Same I just leave 6.1 on Max. Don't touch that dial
steinvakt2 [3 hidden]5 mins ago
Same here. I’m feeling productive enough. If I was more “productive”, I wouldn’t be able to keep up with what I was producing. My attention and cognitive broadband is the bottleneck. I find it hard to believe that people are actually productive using 4 active chat windows simultaneously for 8+ hours a day.
I find QFN on xhigh usually overruns the context window for no good result. If it can do the task, it'll do it quicker on medium. If it can't do the task, xhigh won't change that.
Onavo [3 hidden]5 mins ago
I wish a company named human layer won't write their reports with so much Claudisms.
0c3ca83 [3 hidden]5 mins ago
[flagged]
Onavo [3 hidden]5 mins ago
I think people downvoting you missed the sarcasm.
perching_aix [3 hidden]5 mins ago
On the contrary.
cute_boi [3 hidden]5 mins ago
The irony is the article is very sloppy. Can we at least write it and review it properly..
antipaul [3 hidden]5 mins ago
This is true. I was unfamiliar with this concept (I use codex/claude daily), and am trying to get to the "bottom line".
Oh, it's there at bottom under "my takeaway". But I still didn't get it initially.
Slop Code Bench? Need a Slop Writing Bench too
jampekka [3 hidden]5 mins ago
What's wrong with it? Reads fine to me.
a3w [3 hidden]5 mins ago
There are 36 tests. You can pass 0 or 36 or any number inbetween. Lets give five significant digits of how accurate solving was.
That is not how data physics work. It should not be how data science works.
HN.zip
https://lucumr.pocoo.org/2026/9/7/astra-why/
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
edit: I wish it would let me view the darn screenshot tho :c
I run Paseo as an orchestrator with agent profiles. "Distinguished Member of Technical Staff" -> Fable, plan mode. "End-User Customer Advocate" -> Astra, plan mode. A few others. Basically, these agents (with profile prompts) can only be "consulted" for input on design elements; it's OK if they produce a bit of verbosity here. This has worked fantastically for me, really awesome to see a self-constructed team own the work end-to-end.
A "Director of Engineering" (6.1 Sol or Opus 5.5) also in plan mode organizes the team for the task and then owns coordination and instruction. You can think of this in essence like a "model router".
Coding is handled by Haiku 5.5/GPT 6 Luna/DeepSeek v4.1 Flash. One "Principal Engineer" Sonnet 5.5 or 6.1 Sol for some tasks.
You may be larping for no additional benefit vs just using Opus 5.5 and just asking it to implement a feature.
This also helps with context growth since the sub-agents do the heavy lifting (e.g. search, read docs) and just provide the key bits back to the lead.
Is it "larping"? I ran an eval today on a contract ingest and extract agent I'm working on. Changing the <role> fragment of the prompt lifted the results by +7 points by simply changing the text to the effect of "You are a senior partner at a distinguished legal firm that focuses on reading licensing contracts..." and some bits about the content the agent is reviewing + objective.
My conclusion: the agents like to LARP.
https://xxcancel.com/tenobrus/status/2096636223414808719
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
And https://github.com/gabeorlanski/scb-problems/
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
That’s a little telling of the state of ai atm though isn’t it? It’s all pretty similar. The latest Opus/GPT/Gemini/Deepseek/Kimi on medium does indeed do the job
Though, it has been a while since I've used 5.5.
But for webdev, deep research, code review, and a few other tasks I find the current 6.1 sol better.
But I use both. And I use codex more.
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)
Recent evaluation result.
I find QFN on xhigh usually overruns the context window for no good result. If it can do the task, it'll do it quicker on medium. If it can't do the task, xhigh won't change that.
Oh, it's there at bottom under "my takeaway". But I still didn't get it initially.
Slop Code Bench? Need a Slop Writing Bench too
That is not how data physics work. It should not be how data science works.