If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.

The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.

Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.

An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.

The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.

If it weren’t OAI and Anthropic, it would be others.

If it weren’t US companies, it would be others.

It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).

Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.

> If it weren’t US companies, it would be others.

There is something deeply rotten about the US mindset though, them being American is definitely a problem.

> Look at what happened with fossil fuels.

The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900

I remember reading about the development of neural networks in a magazine from the 1980s.

Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.

Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.

> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did

Uh? In what universe are openai and anthropic not shadow groups?

"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."

I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.

Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.

So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.

We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.

Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.

However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.

Blockchain dapps were getting hacked long before AI. Not a signal

I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.

I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.

> people will die.

I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.

Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.

You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.

> Not if the weapon is an indiscriminate one and makes everyone worse off.

Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.

Such a smart guy so late for the party. Thanks to him and others who understand a lot about tech, we gave monster a weapon to destroy us.

I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder

So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.

If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.

Of course, we are. Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.

I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.

"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?

Speak for yourself - I'm in AI heaven.

Programming has never been this awesome.

  “I think [Mark] Zuckerberg and [Elon] Musk and Sam Altman, they understand the risks, and they don’t give a shit. They’d rather have the money. That’s my belief.”

What's risk for people valued close to or over trillion dollars? Risk is meaningless when corporations get too big to fail.

"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.

That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.

As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.

AI winter has arrived.

Said AI "experts" for the last 10 years.

Nobody has said that. The winter will probably come, but it's not at all clear when. First the bubble must burst.

I think we did good inventing it and it's inevitable. Lets use it well.

What is the point of this quote? Or any of his quotes for that matter.

So why did he start to develop it in the 1970s?

Why did he start a research direction that 50 years later "might turn out" to not be a good idea?

That's a relatively silly question, don't you think?

No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.

Sounds like sour grapes. He worked on it for a long time but he doesn't own a trillion dollar company. Missed the boat?

modern day Mary Shelley

> It could help cure diseases, end boring work and give us more free time.

Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.

"Could" does a lot of work in these kinds of claims.

We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.

The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.

> We need to build it as fast as possible because it could cure all disease

Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.

How has AI given you less free time?

I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.

The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.

In short, it takes a LOT more time just to find some sanity now.

AI is shit.

That's the whole point of the site. Did you read it. AI should not cause harm, direct or indirect.

Of course I read it. But my thesis is that one cannot avoid that. Causing harm, whether direct or indirect, is the fundamental nature of AI.

I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.

Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way

I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.

You’re a cultist

How did you come to that conclusion?

If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.

He would still be right and the field practioners would be wrong.

So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?

We should regulate data centers to require fully renewable power (solar/wind/hydro + battery).

To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.

Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.

I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.

But considering what access to knowledge was like before AI came along, I think having AI is better.

We should introduce regulation so only credentialed software engineers are allowed to use AI.

I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.

In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.

Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.

The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.

But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good

I guess capsule network haven't improved ...

It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).

No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.

Same with AI. Thus far we’re better off with it than without it. That may change.

I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.

Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?

(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)

>so far I’m happy we haven’t had another world war

The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.

The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave

... no shit Sherlock

My reader mode says 6–7 minutes, so, yes.

TFA could have been two sentences:

He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.

He believes AI might improve our lives, but governments must make sure it is safe.

Did you look at it? It wasn’t that long.

I personally feel reading the article before commenting affects the purity of my vibe before posting

vibe comments!

We should have not left the trees. You cannot disrupt banking and power if they do not exist. Checkmate!

The ghost of Douglas Adams has entered the chat