Pro-LLM coding agents: look! a working compiler built in a few hours by an agent! this is amazing!

Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless.

Pro: Sure, but we can get the agent to fix that.

Anti: Can you, though? We've seen that the more complex the code base, the worse the agents do. Fixing complex issues in a compiler seems like something the agents will struggle with. Also, if they could fix it, why haven't they?

Pro: Sure, maybe now, but the next generation will fix it.

Anti: Maybe. While the last few generations have been getting better and better, we're still not seeing them deal with this kind of complexity better.

Pro: Yeah, but look at it! This is amazing! A whole compiler in just a few hours! How many millions of hours were spent getting GCC to this state? It's not fair to compare them like this!

Anti: Anthropic said they made a working compiler that could compile the Linux kernel. GCC is what we normally compile the Linux kernel with. The comparison was invited. It turned out (for whatever reason) that CCC failed to compile the Linux kernel when GCC could. Once again, the hype of AI doesn't match the reality.

Pro: but it's only been a few years since we started using LLMs, and a year or so since agents. This is only the beginning!

Anti: this is all true, and yes, this is interesting. But there are so many other questions around this tech. Let's not rush into it and mess everything up.

The PR author had zero understanding why their entirely LLM-generated contribution was viewed so suspiciously.

The article validates a significant point: it is one thing to have passing tests and be able to produce output that resembles correctness - however it's something entirely different for that output to be good and maintainable.

[1] https://github.com/ocaml/ocaml/pull/14369

>Beats me. AI decided to do so and I didn't question it.

Haha that's comedy gold, and honestly a good interview screening situation - you'd instantly pass on the candidate!

He told me "I spent n days to architect the solution"

He shows me claude generated system design .. and then i say ok, I went to review the code. 1hr later i asked why did you repeat the code all over at the end. Dude replies "junk the entire PR it's AI generated"

[0] https://github.com/ocaml/ocaml/pull/14369#issuecomment-35565...

It is not just patience, he is ready to spent a shitload of time explaining basics to strangers. Such an answer would take, I believe would take a very least half an hour to compose, not counting the time you need to read all the relevant discussion to get the context. But yeah, it would be great to have more people like him around.

When contributions are small and tightly human-controlled it's also less likely that potential legal concerns will arise, since it means that any genuinely creative decisions about the code are a lot easier to trace.

(In this case, the AI seems to have ripped off a lot of the work from OxCaml with inconsistent attribution. OxCaml is actually license compatible (and friendly) with Ocaml but obviously any merge of that work should happen on its own terms, not as a side effect of ripoff slop code.)

People aren't prompting LLMs to write good, maintainable code though. They're assuming that because we've made a collective assumption that good, maintainable code is the goal then it must also be the goal of an LLM too. That isn't true. LLMs don't care about our goals. They are solving problems in a probabilistic way based on the content of their training data, context, and prompting. Presumably if you take all the code in the world and throw it in mixer what comes out is not our Platonic ideal of the best possible code, but actually something more like a Lovecraftian horror that happens to get the right output. This is quite positive because it shows that with better prompting+context+training we might actually be able to guide an LLM to know what good and bad looks like (based on the fact that we know). The future is looking great.

However, we also need to be aware that 'good, maintainable code' is often not what we think is the ideal output of a developer. In businesses everywhere the goal is 'whatever works right now, and to hell with maintainability'. When a business is 3 months from failing spending time to write good code that you can continue to work on in 10 years feels like wasted effort. So really, for most code that's written, it doesn't actually need to be good or maintainable. It just needs to work. And if you look at the code that a lot of businesses are running, it doesn't. LLMs are a step forward in just getting stuff to work in the first place.

If we can move to 'bug free' using AI, at the unit level, then AI is useful. Above individual units of code, like logic, architecture, security, etc things still have to come from the developer because AI can't have the context of a complete application yet. When that's ready then we can tackle 'tech debt free' because almost all tech debt lives at that higher level. I don't think we'll get there for a long time.

Then they're not using the tools correctly. LLMs are capable of producing good clean code, but they need to be carefully instructed as to how.

I recently used Gemini to build my first Android app, and I have zero experience with Kotlin or most of the libraries (but I have done many years of enterprise Java in my career). When I started I first had a long discussion with the AI about how we should set up dependency injection, Material3 UI components, model-view architecture, Firebase, logging, etc and made a big Markdown file with a detailed architecture description. Then I let the agent mode implement the plan over several steps and with a lot of tweaking along the way. I've been quite happy with the result, the app works like a charm and the code is neatly structured and easy to jump into whenever I need to make changes. Finishing a project like this in a couple of dozen hours (especially being a complete newbie to the stack) simply would not have been possible 2-3 years ago.

I'd argue that when the code is part of a press release or corporate blog post (is there even a difference?) by the company that the LLM in question comes from, e.g. Claude's C compiler, then one cannot reasonably assert they were "not using the tools correctly": even if there's some better way to use them, if even the LLM's own team don't know how to do that, the assumption should be that it is unreasonable to expect anyone else to how to do that either.

I find it interesting and useful to know that the boundary of the possible is a ~100kloc project, and that even then this scale of output comes with plenty of flaws.

Know what the AI can't do, rather than what it can. Even beyond LLMs, people don't generally (there's exceptions) get paid for manually performing tasks that have already been fully automated, people get paid for what automation can't do.

Moving target, of course. This time last year, my attempt to get an AI to write a compiler for a joke language didn't even result in the source code for the compiler itself compiling; now it not only compiles, it runs. But my new language is a joke language, no sane person would ever use it for a serious project.

Yep, that's how you get better output from AI. A lot of devs haven't learned that yet. They still see it as 'better autocomplete'.

A good AI development framework needs to support a tail of deprecated choices in the codebase.

Skills are considerable better for this than design docs.

LLMs do not learn. So every new session for them will be rebuilding the world from scratch. Bloated Markdown files quickly exhaust context windows, and agents routinely ignore large parts of them.

And then you unleash them on one code base that's more than a couple of days old, and they happily duplicate code, ignore existing code paths, ignore existing conventions etc.

Finding and fixing that subtle, hard to reproduce bug that could kill your business after 3 years.

But it's not going to be as easy as "just regenerate everything". There are dependencies external to a particular codebase such as long lived data and external APIs.

I also suspect that the stability of the codebase will still matter, maybe even more so than before. But the way in which we define maintainability will certainly change.

Adding Ai generated comments are IMHO some of the most rude uses of Ai.

A slightly sarcastic (or perhaps not so slightly..) mental model of legal conflict resolution is that much of it boils down to throwing lots of content at the opposing side, claiming that it shows that the represented side is right and creating a task for the opposite side to find a flaw in that material. I believe that this game of quantity fits through the whole range from "I'll have my lawyer repeat my argument in a letter featuring their letter head" all the way to paper-tsunamis like the Google-Oracle trial.

Now give both sides access to LLM... I wonder if the legal profession will eventually settle on some format of in-person offline resolution with strict limits to recess and/or limits to word count for both documents and notes, because otherwise conflicts fail to get settled in anyone's lifetime (or won by whoever does not run out of tokens first - come thinking of it, the technogarchs would love this, so I guess this is exactly what will happen barring a revolution)

I am referring to the act of merely pasting the output of a model as a comment.

Have the decency to understand what the LLM is writing and write your own message.

> Here's the AI-written copyright analysis...

I'm not going to spend more time reading than what you have spent writing!

Pretty soon we'll have AIs talking to each other.

I wouldn't call this a fiasco, it reads to me more that being able to create huge amounts of code - whether the end result works well or not - breaks the traditional model of open source. Small contributions can be verified and the merrit-vs-maintenance-effort can at least be assessed somewhat more realistically.

I have no bones in the "vibe coding sucks" vs "vibe coding rocks" discussion and I reading that thread as an outsider. I cannot help but find the PR author's attitude absolutely okay while the compiler folks are very defensive. I do agree with them that submitting a huge PR request without prior discussion cannot be the way forward. But that's almost orthogonal to the question of whether AI-generated code is or is not of value.

If I were the author, I would probably take my 13k loc proof-of-concept implementation and chop it down into bite-size steps that are easy to digest, and try to get them to get integrated into the compiler successively, with being totally upfront about what the final goal is. You'd need to be ready to accept criticism and requests for change, but it should not be too hard to have your AI of choice incorporate these into your code base.

I think the main mistake of the author was not to use vibe coding, it was to dream up his own personal ideal of a huge feature, and then go ahead and single-handedly implement the whole thing without involving anyone from the actual compiler project. You cannot blame the maintainers for not being crazy about accepting such a huge blob.

I struggle to think how someone thinks this is polite. Is politeness to you just not using curse words?

But I am biased. After having lived a number of years in a country where I would say the average understanding of politeness is vastly different from where I've grown up, I've learned that there is just a difference of opinion of what is polite and what isn't. I have probably been affected by that too.

if you will , place yourself in the shoes of the repository maintainer. a random person (with a personal agenda) has popped up trying to sell you a solution (that he doesn't understand) to a problem (that you don't see as problematic). after you spending literal hours patiently explaining why the proposition is not acceptable , this random person still continues attempting to sell his solution.

do you see any impoliteness in the reframed scenario ?

Ah yes. "It's AI I don't care" and "AI has very deep reasoning about code, prove me wrong" are the height of politeness.

"Pro": give me tons of money to keep going this endeavour.

More seriously: if some LLM or other can _assist_ c++ to plain and simple C port...

- Ohh look it can [write small function / do a small rocket hop] but it can't [ write a compiler / get to orbit]!

- Ohh look it can [write a toy compiler / get to orbit] but it can't [compile linux / be reusable]

- Ohh look it can [compile linux / get reusable orbital rocket] but it can't [build a compiler that rivals GCC / turn the rockets around fast enough]

- <Denial despite the insane rate of progress>

There's no reason to keep building this compiler just to prove this point. But I bet it would catch up real fast to GCC with a fraction of the resources if it was guided by a few compiler engineers in the loop.

We're going to see a lot of disruption come from AI assisted development.

But also we've just witnessed LLMs go from being a glorified line auto-complete tool to it writing a C compiler in ~3 years. And I think that's something. And noting how we keep moving the goal post.

You: "but look! It wrote a C compiler!"

> - <Denial despite the insane rate of progress>

Sure, but not by what was actually promised. There may also be fundamental limitations to what the current architecture of LLMs can achieve. The vast majority of LLMs are still based on Transformers, which were introduced almost a decade ago. If you look at the history of AI, it wouldn't be the first time that a roadblock stalled progress for decades.

> But I bet it would catch up real fast to GCC with a fraction of the resources if it was guided by a few compiler engineers in the loop.

Okay, so at that point, we would have proved that AI can replicate an existing software project using hundreds of thousands of dollars of computing power and probably millions of dollars in human labour costs from highly skilled domain experts.

Yeah but the speed of progress can never catch the speed of a moving goalpost!

I still won’t use it while it also matches all the non-functional requirements but you’re free to go and recompile all the software you use with it.

How do you like those coast-to-coast self drives since the end of 2017?

Interesting that people call this "progress" :)

But "reliable, durable, scalable outcomes in adversarial real-world scenarios" is not convincingly demonstrated in public, the asterisks are load bearing as GPT 5.2 Pro would say.

That game is still on, and AI assist beyond FIM is still premature for safety critical or generally outcome critical applications: i.e. you can do it if it doesn't have to work.

I've got a horse in this race which is formal methods as the methodology and AI assist as the thing that makes it economically viable. My stuff is north of demonstrated in the small and south of proven in the large, it's still a bet.

But I like the stock. The no free lunch thing here is that AI can turn specifications into code if the specification is already so precise that it is code.

The irreducible heavy lift is that someone has to prompt it, and if the input is vibes the output will be vibes. If the input is zero sorry rigor... you've just moved the cost around.

The modern software industry is an expensive exercise in "how do we capture all the value and redirect it from expert computer scientists to some arbitrary financier".

You can't. Not at less than the cost of the experts if the outcomes are non-negotiable.

In 1935 the Auburn 851 S/C Speedster hit 100mph

In 1955 the Mercedes-Benz 300 SL Gullwing did 161mph

In 2025 the Yangwang U9 Xtreme hit 308mph

progress is a decaying exponential - Tsiolkovsky's tyranny

Human crews on Mars is just as far fetched as it ever was. Maybe even farther due to Starlink trying to achieve Kessler syndrome by 2050.

The problem is that it is absolutely indiscernible from the Theranos conversation as well…

If Anthropic stopped making lies about the current capability of their models (like “it compiles the Linux kernel” here, but it's far from the first time they do that), maybe neutral people would give them the benefit of the doubt.

For one grifter that happen to succeed at delivering his grandiose promises (Elon), how many grifters will fail?

https://git.sr.ht/~xigoi/hilda

Do we need a c2 wiki page for "sufficiently smart LLM" like we do for https://wiki.c2.com/?SufficientlySmartCompiler ?

How many developers do you think are solving truly novel problems? Most like me are CRUD bunnies.

That's a really nice fictitious conversation but in my experience "anti-ai" people would be prone to say "This is stupid LLM's will never be able to write complex code and attempting to do so is futile". If your mind is open to explore how LLM's will actually write complex software then by definition you are not "anti".

Unless you need a correctly compiled Linux kernel. In that case one gets exhausting real quick.

I don't think those are the only two options, though.

Further, "Getting really good at leveraging AI" is very different to "Getting really good at prompting LLMs".

One is a skill that might not even result in the AI providing any code. The other is a "skill" in much the same way as winning hotdog eating contests is a "skill".

In the latter, even the least-technical user can replace you once they get even halfway decent at min-maxing their agent's input (md files, although I expect we'll switch away from that soon enough to a cohesive and structured UI).

In the former, you had better find some really difficult problems that pay when you solve them.

Either way, I predict a lot of pain and anguish in the near future, for a lot of people. Especially those who expect that prompting skills are an actual "skill".

As someone who leans pro in this debate, I don't think I would make that statement. I would say the results are exactly as we expect.

Also, a highly verifiable task like this is well suited to LLMs, and I expect within the next ~2 years AI tools will produce a better compiler than gcc.

That's what always puts me off: when AI replaces artists, SO and FOSS projects, it can only feed into itself and deteriorate..

We can't do it for all domains, but I believe we can for efficient code.

today's models could be probably already good enough to compose tasks, and evaluate the results.

Building a "better compiler than gcc" is a matter of cutting-age scientific research, not of being able to write good code

Right.

and the "anti" crowd will point to some exotic architecture where it is worse

Yes it will be far easier than if they did it without AI, but should we really call it “produced by AI” at that point?

I think compilers though are a rare case where large scale automated verification is possible. My guess is that starting from gcc, and all existing documentation on compilers, etc. and putting ridiculous amounts of compute into this problem will yield a compiler that significantly improves benchmarks.

I think the pro would tell you that if GCC developers could leverage Opus 4.6, they'd be more productive.

The anti would tell you that it doesn't help with productivity, it makes us less versed in the code base.

I think the CCC project was just a demonstration on what Opus can do now autonomously. 99.9% of software projects out there aren't building something as complex as a Linux compiler.

Maybe one of those companies will come out on top. The others produce garbage in comparison. Capital loves a single throat to choke and doesn't gently pluralise. So of course you buy the best service. And it really can generate any code, get it working, bug free. People unlearn coding on this level. And some day, poof, Microsoft is coming around and having some tiny problem that it can generate a working Office clone. Or whatever, it's just an example.

This technology will never be used to set anyone free. Never.

The entity that owns the generator owns the effective means of production, even if everyone else can type prompts.

The same technology could, in a different political and economic universe, widen human autonomy. But that universe would need strong commons, enforced interoperability, and a cultural refusal to outsource understanding.

And why is this different from abstractions that came before? There are people out there understanding what compilers are doing. They understand the model from top to bottom. Tools like compilers extended human agency while preserving a path to mastery. AI code generation offers capability while dissolving the ladder behind you.

We are not merely abstracting labor. We are abstracting comprehension itself. And once comprehension becomes optional, it rapidly becomes rare. Once it becomes rare, it becomes political. And once it becomes political, it will not be distributed generously.

With so many companies in AI race it is already pretty competitive landscape and it doesnt seem likely to me that any of them can build deep enough moat to come ahead.

What an amazing tech. And look, the CEOs are promising us a good future! Maybe we can cool the datacenters with Brawndo. Let me ask chat if that is a good idea.

I am "pro" in the sense that I believe that LLM's are making traditional programming obsolete. In fact there isn't any doubt in my mind.

However, I am "anti" in the sense that I am not excited or happy about it at all! And I certainly don't encourage anyone to throw money at accelerating that process.

Is this what AI psychosis looks like? How can anyone that is a half decent programmer actually believe that English + non-deterministic code generator will replace "traditional" programming?

4GLs are productive yes, but also limited, and still require someone to come up with specs that are both informed by (business) realities and engineering considerations.

But this is also an arena where bosses expect magic to happen when people are involved; just pronounce a new strategy, and your business magically transforms - without any of that pesky 'figuring out what to do' or 'aligning stakeholders' or 'wondering what drugs the c-suite is doing'. Let LLMs write the specs!

That's a valid take. The problem is that there are, at this time, so many valid takes that it's hard to determine which are more valid/accurate than the other.

FWIW, I think this is more insightful than most of the takes I've seen, which basically amount to "side-1: we're moving to a higher level of abstraction" and "side-2: it's not higher abstraction, just less deterministic codegen".

How do you solve a problem you refuse to define explicitly? We end up with these Goodhart's Law solutions: they hit all of the required goals and declare victory, but completely fail in every reasonable metric that matters. Which I guess is an approach you make when you are selling agents by the token, but I don't see why anyone else is enamored with this approach.

“It will get better, and then we will use it to make many of you unemployed”

Colour-me-shocked that swathes of this industry might have an issue with that.

> Anti-LLM coding agents: it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless.

Pro-LLM: Read the freaking article, it's not that long. The compiler made a mistake in an area where only two compilers exist that are up to the task: Linux Kernel.

It turns out that isn't true in all instances, as this article demonstrates. I'm not nearly expert enough to be able to decide if that error was simple, stupid, irrelevant, or whatever. I can make a call on whether it successfully compiled the Linux kernel: it did not.

https://www.astralcodexten.com/p/if-its-worth-your-time-to-l...

The freaking article omits several issues in the "compiler". My bet is because they didn't actually challenged the output of the LLM, as it usually happens.

If you go to the repository, you'll find fun things, like the fact that it cannot compile a bunch of popular projects, and that it compiles others but the code doesn't pass the tests. It's a bit surprising, specially when they don't explain why those failures exist (are they missing support for some extensions? any feature they lack?)

It gets less surprising, though, when you start to see that the compiler doesn't actually do any type checking, for example. It allows dereferences to non-pointers. It allows calling functions with the wrong number of arguments.

There's also this fantastic part of the article where they explain that the LLM got the code to a point where any change or bug fix breaks a lot of the existing tests, and that further progress is not possible.

Then the fact that this article points out that the kernel doesn't actually link. How did they "boot it"? It might very well be possible that it crashed soon after boot and wasn't actually usable.

So, as usual, the problem here is that a lot of people look at LLM outputs and trust what they're saying they achieved.

[1] There are around 8k programming languages that are used (or were used) in practice (that is, they were deemed better than existing ones in some aspects) and there are around 50 million programmers. I use it to estimate how many people did something, which is objectively better than existing products.

> it's not a working compiler, though. And it doesn't matter how few hours it took, because it doesn't work. It's useless.

It works. It's not perfect, but anthropic claims to have successfully compiled and booted 3 different configurations with it. The blog post failed to reproduce one specific version on one specific architecture. I wish anthropic gave us more information about which kernel commits succeeded, but still. Compare this to years that it took for clang to compile the kernel, yet people were not calling that compiler useless.

If anyone thinks other compilers "just work", I invite them to start fixing packages that fail to build in nixos after every major compiler change, to get a dose of real world experience.

"The source code of gcc is available online"

Honestly? I think if we as a society could trust our leaders (government and industry) to not be total dirtbags the resistance to AI would be much lower.

Like imagine if the message was “hey, this will lead to unemployment, but we are going to make sure people can still feed their families during the transition, maybe look in to ways to support subsidies retraining programs for people whose jobs have been impacted .” Seems like a much more palatable narrative than, “fuck you pleb! go retrain as a plumber or die in a ditch. I’ll be on my private island counting the money I made from destroying your livelihood.”

First, remember when we had LLMs run optimisation passes last year? Alphaevolve doing square packing, and optimising ML kernels? The "anti" crowd was like "well, of course it can automatically optimise some code, that's easy". And things like "wake me up when it does hard tasks". Now, suddenly when they do hard tasks, we're back at "haha, but it's unoptimised and slow, laaame".

Second, if you could take 100 juniors, 100 mid level devs and 100 senior devs, lock them in a room for 2 weeks, how many working solutions that could boot up linux in 2 different arches, and almost boot in the third arch would you get? And could you have the same devs now do it in zig?

The thing that keeps coming up is that the "anti" crowd is fighting their own deamons, and have kinda lost the plot along the way. Every "debate" is about promisses, CEOs, billions, and so on. Meanwhile, at every step of the way these things become better and better. And incredibly useful in the right hands. I find it's best to just ignore the identity folks, and keep on being amazed at the progress. The haters will just find the next goalpost and the next fight with invisible entities. To paraphrase - those who can, do, those who can't, find things to nitpick.

Codex frustratingly failed at refactoring my tests for me the other day, despite me trying many, many prompts of increasing specificity. A task a junior could've done

Am I saying "haha it couldn't do a junior level task so therefor anything harder is out of reach?" No, of course not. Again, it's not a human. The comparison is irrelevant

Calculators are superhuman at arithmetic. Not much else, though. I predict this will be superhuman at some tasks (already is) and we'll be better at others

Burn everything, because if you don’t, you will get left behind and, maybe, just maybe, in 2 years when it’s good enough, maybe… after hoovering up all the money, IP and domain expertise for free, and you’ve burnt all your money & sanity prompting and cajoling it to a semi working solution for a problem you didn’t really have in the first place, it will dump you at the back of the unemployment line. All hail the AI! Crazy times.

In the meantime please enjoy targeted scams, ever increasing energy prices, AI content farms, hardware shortages, and endless, endless slop.

When humans architect anything - ideas, buildings, software or ice cream sundaes, we make so many little decisions that affect the overall outcome, we don’t even know or think about it! Too many sprinkles and sauce and it will be too sweet and hard to eat. We make those decisions based on both experience and imagination. Watch a small child making one to see the perfect human intersection of these two things at play. The LLM totally lacks the imagination part, except in the worst possible ways. It’s experience includes all sorts of random internet garbage that can sound highly convincing even to domain experts. Now it’s training set is being further expanded with endless mountains of more highly impressive sounding garbage.

It was obvious to me with the first image gen models how incredibly impressive it was to see an image gradually forming from the computer based on nothing but my brief text input but also how painfully limited the technology would always be. After days and days of early obsessive image generation, I was no better as an artist than when I began! Everything also kind of looked the same as well?

As incredible as it was, it was nothing more than a massively complicated, highly advanced parlour trick. A futuristic, highly powerful pattern generator. Nothing has changed my mind at all. All that’s happened is we’ve seen the worst tricksters, shysters and con artists jump on a very dangerous bandwagon to hell and try and whip us less compliant souls onboard.

Lots of things follow patterns, the joy in life, for me, is discovering the patterns, exploring them and developing new unique and interesting patterns.

I’ve yet to encounter a bandwagon worth joining anyway, maybe this will be the one that leaves me behind and i’ll be forced to retire on cartoon gorilla NFTs and tulip farming?

Second depends. If you told them to pretrain for writing C compiler however long it takes, I could see a smaller team doing it in a week or two. Keep in mind LLMs pretrain on all OSS including GCC.

> Meanwhile, at every step of the way these things become better and better.

Will they? Or do they just ingest more data and compute?[1] Again, time will tell. But to me this seems more like speed-running into an Idiocracy scenario than a revolution.[2]

I think this will turn out another driverless car situation where last 1% needs 99% of the time. And while it might happen eventually it's going to take extremely long time.

[1] Because we don't have much more computing jumps left, nor will future data be as clean as now.

[2] Why idiocracy?

Because they are polluting their own corpus of data. And by replacing thinking about computers, there will be no one to really stop them.

We'll equalize the human and computer knowledge by making humans less knowledgeable rather than more.

So you end up in an Idiocracy-like scenario where a doctor can't diagnose you, nor can the machine because it was dumbed down by each successive generation, until it resembles a child's toy.

It's a system based on LLMs.

> AlphaEvolve, an evolutionary coding agent powered by large language models for general-purpose algorithm discovery and optimization. AlphaEvolve pairs the creative problem-solving capabilities of our Gemini models with automated evaluators that verify answers, and uses an evolutionary framework to improve upon the most promising ideas.

> AlphaEvolve leverages an ensemble of state-of-the-art large language models: our fastest and most efficient model, Gemini Flash, maximizes the breadth of ideas explored, while our most powerful model, Gemini Pro, provides critical depth with insightful suggestions. Together, these models propose computer programs that implement algorithmic solutions as code.

What you said is:

    > First, remember when we had LLMs run optimisation passes last year? Alphaevolve doing square packing

    First, remember the fish intelligence competition last year? Rossie jumped through a hula hoop.

Alpha Evolve is made from LLMs, but it's not the only part. If anything, it needs a genetic algo component. LLMs generally don't evolve.

Also, why are you focusing on AlphaEvolve? I made two other points you haven't addressed.

I personally hope that that happens, but I doubt it will. Note also that processors still continued to improve even without Dennard Scaling due to denser, better optimized onboard caches, better branch prediction, and more parallelism (including at the instruction level), and the broader trend towards SoCs and away from PCB-based systems, among other things. So at least by analogy, it's not impossible that even with that conjectured roadblock, Big AI could still find room for improvement, just at a much slower rate.

But current LLMs are thoroughly compelling, and even just continued incremental improvements will prove massively disruptive to society.

At the same time - you could direct Claude to review the register spilling code and the linker code of both LLVM/gcc for potential improvements to CCC and you will see improvements. You can ask it not to copy GPL code verbatim but to paraphrase and tell it it can rip code from LLVM as long as the licenses are preserved. It will do it.

You might only see marginal improvements without spending another $100K on API calls. This is about one of the hardest projects you could ask it to bite off and chew on. And would you trust the compiler output yet over GCC or LLVM?

Of course not.

But I wager, that if you _started_ with the LLVM/gcc codebases and asked it to look for improvements - it might be surprising to see what it finds.

Both sides have good arguments. But this could be a totally different ball game in 2, 5 and 10 years. I do feel like those who are most terrified by it are those whose identity is very much tied to being a programmer, and seeing the potential for their role to be replaced and I can understand that.

Me personally - I'm relieved I finally have someone else to blame and shout at rather than myself for the bugs in the software I produce. I'm relieved that I can focus now on the more creative direction and design of my personal projects (and even some work projects on the non-critical paths) and not get bogged down in my own perfectionism with respect to every little component until reaching exhaustion and giving up.

And I'm fascinated by the creativity of some of the projects I see that are taking the same mindset and approach.

I was depressed by it at first. But as I've experimented more and more, I've come to enjoy seeing things that I couldn't ever have achieved even with 100 man years of my own come to fruition.

But lying and hype is baked into the DNA of AI booster culture. At this point it can be safely assumed anything short of right-here-right-now proof is pure unfettered horseshit when coming from anyone and everyone promoting the value of AI.

I learned about it from HackerNews and ChatGPT.

It seemed pretty unambiguous to me from the blog post that they were saying the kernel could boot on all three arch's, but clearly that's not true unless they did some serious hand-waving with kernel config options. Looking closer in the repo they only show a claimed Linux boot for RISC-V, so...

[0]: https://www.anthropic.com/engineering/building-c-compiler - "build a bootable Linux 6.9 on x86, ARM, and RISC-V."

[1]: https://github.com/anthropics/claudes-c-compiler/blob/main/B... - only shows a test of RISC-V

In the specific case of __jump_table I would even guess there was some work in getting the Clang build working.

I'm not dissing CCC here, rather I'm impressed with how much speed is squeezed out by GCC out of what is assumed to be already an intrinsically fast language.

The primatives are directly related to the actual silicon. A function call is actually going to turn into a call instruction (or get inlined). The order of bytes in your struct are how they exist in memory, etc. A pointer being dereferenced is a load/store.

The converse holds as well. Interpreted languages are slow because this association with the hardware isn't the case.

When you have a poopy compiler that does lots of register shuffling then you loose this association.

Specifically the constant spilling with those specific functions functions that were the 1000x slowdown, makes the C code look a lot more like Python code (where every variable is several dereference away).

> The compiler did its job fine

> Where CCC Succeeds Correctness: Compiled every C file in the kernel (0 errors)

I don't think that follows. It's entirely possible that the compiler produced garbage assembly for a bunch of the kernel code that would make it totally not work even if it did link. (The SQLite code passing its self tests doesn't convince me otherwise, because the Linux kernel uses way more advanced/low-level/uncommon features than SQLite does.)

Whenever I've done optimisation (e.g. genetic algorithms / simulated annealing) before you always have to be super careful about your objective function because the optimisation will always come up with some sneaky lazy way to satisfy it that you didn't think of. I guess this is similar - their objective was to compile valid C code and pass some tests. They totally forgot about not compiling invalid code.

I'd like to see someone disagree with the following:

Building a C compiler, targeting three architectures, is hard. Building a C compiler which can correctly compile (maybe not link) the modern linux kernel is damn hard. Building a C compiler which can correctly compile sqlite and pass the test suite at any speed is damn hard.

To the specific issues with the concrete project as presented: This was the equivalent of a "weekend project", and it's amazing

So what if some gcc is needed for the 16-bit stuff? So what if a human was required to steer claude a bit? So what if the optimizing pass practically doesn't exist?

Most companies are not software companies, software is a line-item, an expensive, an unavoidable cost. The amount of code (not software engineering, or architecture, but programming) developed tends towards glue of existing libraries to accomplish business goals, which, in comparison with a correct modern C compiler, is far less performance critical, complex, broad, etc. No one is seriously saying that you have to use an LLM to build your high-performance math library, or that you have to use an LLM to build anything, much in the same way that no one is seriously saying that you have to rewrite the world in rust, or typescript, or react, or whatever is bothering you at the moment.

I'm reminded of a classic slashdot comment--about attempting to solve a non-technical problem with technology, which is doomed to fail--it really seems that the complaints here aren't about the LLMs themselves, or the agents, but about what people/organizations do with them, which is then a complaint about people, but not the technology.

> Building a C compiler which can correctly compile (maybe not link) the modern linux kernel is damn hard.

Is it correctly compiling it? Several people have pointed out that the compiler will not emit errors for clearly invalid code. What code is it actually generating?

> Building a C compiler which can correctly compile sqlite and pass the test suite at any speed is damn hard.

It's even harder to have a C compiler that can correctly compile SQLite and pass the test suite but then the SQLite binary itself fails to execute certain queries (see https://github.com/anthropics/claudes-c-compiler/issues/74).

> which, in comparison with a correct modern C compiler, is far less performance critical, complex, broad, etc.

That code might be less complex for us, but more complex for an LLM if it has to deal with lots of domain-specific context and without a test suite that has been developed for 40 years.

Also, if the end result of the LLM has the same problem that Anthropic concedes here, which is that the project is so fragile that bug fixes or improvements are really hard/almost impossible, that still matters.

> it really seems that the complaints here aren't about the LLMs themselves, or the agents, but about what people/organizations do with them, which is then a complaint about people, but not the technology

It's a discussion about what the LLMs can actually do and how people represent those achievements. We're point out that LLMs, without human supervision, generate bad code, code that's hard to change, with modifications specifically made to address failing tests without challenging the underlying assumptions, code that's inconsistent and hard to understand even for the LLMs.

But some people are taking whatever the LLM outputs at face value, and then claiming some capabilities of the models that are not really there. They're still not viable for using without human supervision, and because the AI labs are focusing on synthetic benchmarks, they're creating models that are better at pushing through crappy code to achieve a goal.

Imagine five years ago saying that you could have a general purpose AI write a c compiler that can handle the Linux kernel, by itself, from scratch for $20k by writing a simple English prompt.

That would have been completely unbelievable! Absurd! No one would take it seriously.

And now look at where we are.

You’re very conveniently ignoring the billions in training and that it has practically the whole internet as input.

And that’s where my suspicion stems from.

An equivalent original human piece of work from an expert level programmer wouldn’t be able to do this without all the context. By that I mean all the all the shared insights, discussion and design that happened when making the compiler.

So to do this without any of that context is likely just very elaborate copy pasta.

Just because we're here doesn't mean we're getting to AGI or software developers begging for jobs at Starbucks

New code generation / modification I think we're hitting a point of diminishing returns and they're not going to improve much here

The limitation is fundamentally that they can only be as good as the detail in the specs given, or the test harnesses provided to them. Any detail left out they're going to make up, and hopefully it's what you want (often it's not!). If you make the specs detailed enough so that there's no misunderstanding possible: you've just written code, what we already do today

Code optimization I think they'll get quite a bit better. If you give them GCC it's probable they'll be able to improve upon it

I wanted to refactor a bunch of tests in a TypeScript project the other day into a format similar to table driven tests that are common in Golang, but seemingly not so much in TypeScript. Vitest has specific syntax affordances for it, though

It utterly failed at the task. Tried many times with increasing specificity in my prompt, did one myself and used it as an example. I ended up giving up and just doing it manually

This will be true for next 2 years, 4 years, next decade, few decades. Until the state of the art ML paradigm remains language models.

The assembler is harder than it looks. It needs to know the exact binary encoding of every instruction for the target architecture. x86-64 alone has thousands of instruction variants with complex encoding rules (REX prefixes, ModR/M bytes, SIB bytes, displacement sizes). Getting even one bit wrong means the CPU will do something completely unexpected.

The linker is arguably the hardest. It has to handle relocations, symbol resolution across multiple object files, different section types, position-independent code, thread-local storage, dynamic linking and format-specific details of ELF binaries. The Linux kernel linker script alone is hundreds of lines of layout directives that the linker must get exactly right."

I worked on compilers, assemblers and linkers and this is almost exactly backwards

This explanation confused me too:

  Each individual iteration: around 4x slower (register spilling)
  Cache pressure: around 2-3x additional penalty (instructions do not fit in L1/L2 cache)
  Combined over a billion iterations: 158,000x total slowdown

Supporting linker scripts is marginally harder, but having manually written compilers before, my experience is the exact opposite of yours.

I thought it was just the compiler that Anthropic produced.

I feel it should for a specific platform/target, but I don't know if it did.

Writing a linker is still a lot of work, so if their original $20k cost of production did not include a linker I'd be less impressed.

Which raises the question, did CC also produce its own pre-processor or just use one of the many free ones?

These kinds of tasks are relatively easy for LLMs, they’re operating in a solved design space and recombining known patterns. It looks impressive to us because writing a compiler from scratch is difficult and time consuming for a human, not because of the problem itself.

That doesn’t mean LLMs aren’t useful, even if progress plateaued tomorrow, they’d still be very valuable tools. But building yet another C compiler or browser isn’t that compelling as a benchmark. The industry keeps making claims about reasoning and general intelligence, but I’d expect to see systems producing genuinely new approaches or clearly better solutions, not just derivations of existing OSS.

Instead of copying a big project, I'd be more impressed if they could innovate in a small one.

That said, I think the framing of "CCC vs GCC" is wrong. GCC has had thousands of engineer-years poured into it. The actually impressive thing is that an LLM produced a compiler at all that handles enough of C to compile non-trivial programs. Even a terrible one. Five years ago that would've been unthinkable.

The goalpost everyone should be watching isn't "can it match GCC" — it's whether the next iteration closes that 158,000x gap to, say, 100x. If it does, that tells you something real about the trajectory.

It says that a nested query does a large number of iterations through the SQLite bytecode evaluator. And it claims that each iteration is 4x slower, with an additional 2-3x penalty from "cache pressure". (There seems to be no explanation of where those numbers came from. Given that the blog post is largely AI-generated, I don't know whether I can trust them not to be hallucinated.)

But making each iteration 12x slower should only make the whole program 12x slower, not 158,000x slower.

Such a huge slowdown strongly suggests that CCC's generated code is doing something asymptotically slower than GCC's generated code, which in turn suggests a miscompilation.

I notice that the test script doesn't seem to perform any kind of correctness testing on the compiled code, other than not crashing. I would find this much more interesting if it tried to run SQLite's extensive test suite.

It could have spotted out GCC source code verbatim and matched its performance.

If I had GCC and was asked for a C compiler I would just provide GCC..

I don't think that's a valid explanation. If something takes 8x as long then if you do it a billion times it still takes 8x as long. Just now instead of 1 vs 8 it's 1 billion vs 8 billion.

I'd be curious to know what's actually going on here to cause a multiple order of magnitude degradation compared to the simpler test cases (ie ~10x becomes ~150,000x). Rather than I-cache misses I wonder if register spilling in the nested loop managed to completely overwhelm L3 causing it to stall on every iteration waiting for RAM. But even that theory seems like it could only account for approximately 1 order of magnitude, leaving an additional 3 (!!!) orders of magnitude unaccounted for.

I think there's a lot more to the story here.

Generally after the SSA pass, you convert all of them into register transfer language (RTL) and then do register allocation pass. And for GCC's case it is even more extreme -- You have GIMPLE in the middle that does more aggressive optimization, similar to rustc's MIR. CCC doesn't have all that, and for register allocation you can try to do simple linear scan just as the usual JIT compiler would do though (and from my understanding, something CCC should do at a simple cost), but most of the "hard part" of compiler today is actually optimization -- frontend is mostly a solved problem if you accept some hacks, unlike me who is still looking for an elegant academic solution to the typedef problem.

  typedef int AA;
  
  void foo()
  {
    AA AA;            /\* OK - define variable AA of type AA */
    int BB = AA * 2;  /\* OK - AA is just a variable name here \*/
  }

  void bar()
  {
    int aa = sizeof(AA), AA, bb = sizeof(AA);
  }

I don't know off the top of my head whether there's a parser framework that makes this parse "straightforward" to express.

Then, only the parser needs to be context sensitive, for the A* B; construct which is either a no-op multiplication (if A is a variable) or a variable declaration of a pointer type (if A is a type)

- Old Russian proverb.

The article is clear about its limitations. The code README opens by saying “don’t use this” which no research paper I know is honest enough to say.

As for hype, it’s less hyped than most university press releases. Of course since it’s Anthropic, it gets more attention than university press.

I think the people most excited are getting ahead of themselves. People who aren’t impressed should remember that there is no C compiler written in Rust for it to have memorized. But, this is going to open up a bunch of new and weird research directions like this blog post is beginning to do.

By evaluating the objective (successful compilation) in a loop, the LLM effectively narrows the problem space. This is why the code compiles even when the broader logic remains unfinished/incorrect.

It’s a good example of how LLMs navigate complex, non-linear spaces by extracting optimal patterns from their training data. It’s amazing.

p.s. if you translate all this to marketing jargon, it’ll become “our LLM wrote a compiler by itself with a clean room setup”.

Edit: typo

1. In the real world, for a similar task, there are little reasons for: A) not giving the compiler access to all the papers about optimizations, ISAs PDFs, MIT-licensed compilers of all the kinds. It will perform much better, and this is a proof that the "uncompressing GCC" is just a claim (but even more point 2).

2. Of all the tasks, the assembler is the part where memorization would help the most. Instead the LLM can't perform without the ISA documentation that it saw repreated infinite number of times during pre-training. Guess what?

3. Rust is a bad language for the test, as a first target, if you want an LLM-coded Rust C compiler, and you have LLM experience, you would go -> C compiler -> Rust port. Rust is hard when there are mutable data structures with tons of references around, and a C compiler is exactly that. To compose complexity from different layers is an LLM anti pattern that who worked a lot with automatic programming knows very well.

4. In the real world, you don't do a task like that without steering. And steering will do wonders. Not to say that the experiment was ill conceived. The fact is that the experimenter was trying to show a different point of what the Internet got (as usually).

"The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."

Large language models and small language models are very strong for solving problems, when the problem is narrow enough.

They are above human average for solving almost any narrow problem, independent of time, but when time is a factor, let's say less than a minute, they are better than experts.

An OS kernel is exactly a problem, that everyone prefers to be solved as correct as possible, even if arriving at the solution takes longer.

The author mentions stability and correctness of CCC, these are properties of Rust and not of vibe coding. Still impressive feat of claude code though.

Ironically, if they populated the repo first with objects, functions and methods with just todo! bodies, be sure the architecture compiles and it is sane, and only then let the agent fill the bodies with implementations most features would work correctly.

I am writing a program to do exactly that for Rust, but even then, how the user/programmer would know beforehand how many architectural details to specify using todo!, to be sure that the problem the agent tries to solve is narrow enough? That's impossible to know! If the problem is not narrow enough, then the implementation is gonna be a mess.

Having LLM generates a first complete iteration of a C compiler in rust is super useful if the code is of good enough quality that it can be maintained and improved by humans (or other AIs). It is (almost) completely useless otherwise.

And that is the case for most of today's code generated by AIs. Most of it will still have to be maintained by humans, or at least a human will ultimately be responsible for it.

What i would like to see is whether that C compiler is a horrible mess of tangled spaghetti code with horrible naming. Or something with a clear structure, good naming, and sensible comments.

Anthropic: "No."

While I agree that the technology behind this is impressive, the biggest issue is license infringement. Everyone knows there's GPL code in the training data, yet there's no trace of acknowledgment of the original authors.

What’s the big deal about that?

How do you know that?

At this point AI coding feels like religion. You have to believe in it.

That’s the whole promise to reach AGI that it will be able to improve itself.

I think Anthropic ruined this by releasing it too early would have been way more fun to have seen a live website where you can see it iterating and the progress is making.

This is what I've noticed about most LLM generated code, its about the quality of an undergrad, and I think there's a good reason for this - most of the code its been trained on is of undergrad quality. Stack overflow questions, a lot of undergrad open source projects, there are some professional quality open source projects (eg SqlLite) but they are outweighed by the mass of other code. Also things like Sqllite don't compare to things like Oracle or Sql Server which are proprietary.

I'm surprised that this wasn't possible before with just a bigger context size.

I have cough AI generated an x86 to x86 compiler (takes x86 in, replaces arbitrary instructions with functions and spits x86 out), at first it was horrible, but letting it work for 2 more days it was actually close to only 50% to 60% slowdown when every memory read instruction was replaced.

Now that's when people should get scared. But it's also reasonable to assume that CCC will look closer to GCC at that point, maybe influenced by other compilers as well. Tell it to write an arm compiler and it will never succeed (probably, maybe can use an intermeriadry and shove it into LLVM and it'll work, but at that point it is no longer a "C" compiler).

It would be interesting to compare the source code used by CCC to other projects. I have a slight suspicion that CCC stole a lot of code from other projects.

Nevertheless, the victories continue to be closer to home.

Perhaps that would be a more telling benchmark to evaluate the Claude compiler against.

But they instead made a blog post about how it would cost you twenty thousand dollars to recreate a piece of software that they do not, with a straight face, actually recommend that you use in any capacity beyond as a toy.

[0] I am categorically not talking about anything AI related or anything that is directly a part of their sales funnel. I am talking about a piece of software that just efficiently does something useful. GCC is an example, Everything by voidtools is an example, Wireshark is an example, etc. Claude is not an example.

I am curious about what results would be for something like a lexer + parser + abstract machine code generator generation for a made up language

Quite well, possibly.

Look, I wasn't even aware of this until it popped up a few days ago on HN, I am not privy to the details of Anthropics engineers in general, or the specific engineer who curated this marathon multi-agent dev cycle, but I can tell you how anyone familiar with compilers or programming language development will proceed:

1. Vibe an IL (intermediate language) specification into existence (even if it is only held in RAM as structures/objects)

2. Vibe some utility functions for the IL (dump, search, etc)

3. Vibe a set of backends, that take IL as input and emit ISA (Instruction Set Architecture), with a set of tests for each target ISA

4. Vibe a front-end that takes C language input and outputs the IL, with a set of tests for each language construct.

(Everything from #2 onwards can be done in parallel)

I have no reason to believe that the engineer who vibe-coded CCC is anything other than competent and skillful, so lets assume he did at least the above (TBH, he probably did more)[1].

This means that CCC has, in its code, everything needed to vibe a never-before-seen ISA, given the ISA spec. It also means it has everything needed to support a new front-end language as long as it is similar enough to C (i.e. language constructs can map to the IL constructs).

So, this should be pretty easy to expand on, because I find it unlikely that the engineer who supervised/curated the process would be anything less than an expert.

The only flaw in my argument is that I am assuming that effort from CC was so large because it did the src -> IL -> ISA route. If my assumption is wrong, it might be well-nigh impossible to add support for a new ISA.

------------------------------

[1] When I agreed to a previous poster on a previous thread that I can recreate the functionality of CCC for $20k, these are the steps I would have followed, except I would not have LLM-generated anything.

I am pretty sure everybody agrees that this result is somewhere between slop code that barely works and the pinnacle of AI-assisted compiler technology. But discussions should not be held from the extreme points. Instead, I am looking for a realistic estimation from the HN community about where to place these results in a human context. Since I have no experience with compilers, I would welcome any of your opinions.

I offered to do it, but without a deadline (I work f/time for money), only a cost estimation based on how many hours I think it should take me: https://news.ycombinator.com/item?id=46909310

The poster I responded to had claimed that it was not possible to produce a compiler capable of compiling a bootable Linux kernel within the $20k cost, nor for double that ($40k).

I offered to do it for $40k, but no takers. I initially offered to do it for $20k, but the poster kept evading, so I settled on asking for the amount he offered.

We act so superior to LLMs but I'm very unimpressed with humanity at this stage.

- Deal with legacy code from day one.

- Have mess of a codebase that is most likely 10-20x the amount of LOC compared to human code

- Have your program be really slow and filled with bugs and edge cases.

This is the battlefield for programmers. You either just build the damn thing or fix bugs for the next decade.

/s

This is actually a nice case study in why agentic LLMs do kind of think. It's by no means the same code or compiler. It had to figure out lots and lots of problems along the way to get to the point of tests passing.

Why the sarcasm tag? It is almost certainly trained on several compiler codebases, plus probably dozens of small "toy" C compilers created as hobby / school projects.

It's an interesting benchmark not because the LLM did something novel, but because it evidently stayed focused and maintained consistency long enough for a project of this complexity.