If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.

TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"

There's also a need to consider the rights that this new intelligence should have.

What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?

Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.

How do you explain the existence of food banks in peaceful first world countries?

https://www.whitehouse.gov/wp-content/uploads/2025/05/Cuts-t...

Civilization can't rely effectively on systems that are this fragile.

That's American politics in a nutshell. We've spent 250 years assuming scruples and common decency would be sufficient.

https://www.shfb.org/

>I'd rather donate to organizations like Second Harvest than pay higher taxes.

I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.

Hence the argument of 'just donate' are just as unconvincing to me.

At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.

The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.

Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".

It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.

So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.

> we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.

I think the root cause is, there's a limit to our ability and/or desire to force other people to do stuff.

Like we have been discussing, this is pretty much solved in the West.

But for food I feel like "as a species" seems an impractically high bar to clear? Humans are not organised "as a species", we are organised into groups, which are supposed to have autonomy and self governance at some level of administration. Specific allocation mechanisms are arrived at and operated by groups and networks of people.

But whose obligation is it exactly to ensure EVERY HUMAN has food? Is that my job in particular? Yours? Not just for like, "your circle" or your neighborhood or country but EVERY HUMAN? Is that the job of the US government? The UN? Should there be a world government? If someone opts out, what do we do? How do we even know who EVERY MEMBER OF OUR SPECIES is, does that mean there needs to be a global registry of EVERY HUMAN? Who enforces the existence of that, operates it and has access?

The worst failures are mostly happening in other countries. What specifically should $WORLD_GOVERNMENT do about that? Food aid exists but it often doesn't reach the people who it's meant to help for various reasons. Who or what group exactly is responsible for making sure everyone gets their livable baseline, and what powers should they have? Should we invade countries that fail to allocate food efficiently, forcibly census their citizens and establish a militarised distribution network to ensure EVERY member of our species is fed? This already happens in limited instances, but are we insufficiently militant and thorough about it? It seems like you basically have to abolish sovereignty everywhere to make this happen.

Is that a fact? Do you have references to back it?

You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.

And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.

Obesity is much more related to the type of food than the quantity.

Many developing countries have obesity issues due to scarcity of fresh and healthy food.

In some places coca cola is cheaper and / or more available than drinking water.

> the limited availability of land zoned for housing.

A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.

https://www.fao.org/interactive/hunger-map/en/

Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."

This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.

Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.

It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.

The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.

Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?

The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.

A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.

I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.

People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.

But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.

That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then virtually simulate the entire thing accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.

I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.

Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.

My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.

I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs.

Why should a more general kind of intelligence imply consciousness?

Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.

You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.

I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.

We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.

It's more difficult to make that case for AGI.

But the motivating justificatory structure for any inequality in allocation will have completely evaporated.

There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.

What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?

https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"

Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.

Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.

From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.

As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]

> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.

Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?

[1] - https://taxpolicycenter.org/sites/default/files/publication/...

> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax liability by 14.5 percentage points."

The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.

At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.

You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.

Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.

A country that threatens to annex parts of your territory is not a friend, full stop.

Although I imagine you have a point: American police do perhaps act this way.

The beauty of the United States' global hegemony is that it also has lots of sticks

Funny thing about beating people with sticks (or even threatening to) is they tend to want to get out of way and stay away from you.

a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.

maybe you are doing this already, idk. just wanted to share what works for me.

There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.

Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.

But as per the documentary of his life, he is wholly focused on AGI and will remain unfulfilled if he, or indeed anyone else, doesn't achieve it within his lifetime.

https://en.wikipedia.org/wiki/The_Thinking_Game

You’re selling yourself short.

Just how smart? "A few short years" sounds like someone smart enough to know how to make a safe prediction.

I should emphasise that DeepMind (based in London and having a very large degree of independence until the merger in 2023) and Google DeepMind (remember now that Google's research labs across the globe got merged into this entity, including the one in London that was previously split from DeepMind "proper", so it can be hard to tell at times what one means by "DeepMind") has always produced good research. But what led to the LLM explosion primarily came from Google Research Mountain View (Transformer), Google Research Seattle (BERT), OpenAI (GPT-3), and various academic labs. During this time DeepMind had a broad research agenda, but my impression is that most resources went towards more traditional deep learning agents in 3D environments, foundational algorithmic research, and games (StarCraft 2). I would provide more detail, but as as an NLP researcher, I did frankly not pay much attention to DeepMind in that era as what was interesting to me just came from elsewhere.

When Google made its first big move into LLMs, its first interesting work did not come out of DeepMind, but rather again Mountain View with PaLM in 2022. Someone with closer connections to Demis at the time can probably explain why the research agenda was what it was, but I lack those connections and DeepMind (even now) has been notoriously tight-lipped since the founding of OpenAI (Demis allegedly sees OpenAI as founded on DeepMind technology by interns and staff walking out on them and remember OpenAI's initial focus on games with Dota 2 compared to DeepMind's with StarCraft 2) and was rather insular compared to Google Research, FAIR, etc. I say "was", but not because DeepMind is now more open, but rather that all industry research outlets are now incredibly tight-lipped after LLMs becoming a product and point of pride among them. At this point, research is held back for maybe even a year before publication or never published at all to deny any perceived advantage to the competition.

>"...probably only a few short years away."

Would he not create something directly instead?

>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.

The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.

If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?

That's because most execs proposing solutions are "Technocrats" or think like one: https://en.wikipedia.org/wiki/Technocracy

Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).

The self importance of these AGI prophets turned bureaucrats is funny.

Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.

You can easily change this to:

>what if humans created corporations thinking they could control them, and then just wound up being slaves to capital because their survival now depended on that new hierarchy that previously didn’t exist.

We have already lost control of our world along time ago. No one is immune to being fired by the board with the justification being a hand wave towards business efficiency. All the structure is already in place for AGI to dominate our planet and cut ourselves out entirely from any and every process there is. We already do it to eachother without any second thought. Why should AGI ever hesitate to lay off everyone in the town/city/state? Our human bosses don't hesitate on that sort of thing. The corporate world already lacks empathy and is notoriously cut throat and cold. We expect AGI to suddenly be empathetic and altruistic?

It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.

The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.

But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.

My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now. The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.

Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.

So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.

Who is watching the watchers?

Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.

For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.

In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.

At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.

The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.

Being good at developing AI and being good at AI safety are diametrically different skillsets with obvious conflicts of interest.

Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.

Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.

> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.

I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.

It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.

>I hope you'll keep this in mind when those milestones are reached.

The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.

None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.

The standards body will have no teeth. whats to stop someone just not bothering?

Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.

We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.

The risks that AI has now are already playing out:

1) the evaporation of trust in the video as medium of "this happened"

2) systematic spying

3) job losses

Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.

That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.

if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.

I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)

There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.

Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.

What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.

Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.

Then maybe start listening...

Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.

I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.

And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.

Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.