One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.

Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.

The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.

Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.

I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!

This problem started before AI hit the mainstream. Make the company “more efficient” (i.e. cut costs) by outsourcing to contractors in areas with lower labor costs. Then, remove escalation paths, then remove abilities from them so that can’t do crazy things like refund a customer. Finally, automate the whole thing and be certain there’s no option available that could possibly return money to the customer.

Indeed.

Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.

AI has no problems. Just properly follow proper procedures.

https://www.youtube.com/watch?v=gEyFH-a-XoQ

Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.

But all we did was ask the AI for world peace. How could that possibly go wrong?

I am in no way making the case for this and can understand the million ways in this can go wrong, but my experience with AI so far has been to make things far LESS Kafkaesque, because it allows you to insert a "human" in the loop far more than our previous systems. Hand rolled software had to boil down to hand written conditionals that tried to account for edge cases, but if you find yourself in an edge case it didn't account for you often just hit a wall. An agent in the loop can immediately understand "oh I see how this doesn't exactly fit into that category but this is clearly ABC."

ah yes kafkaesque. the one with the bug

Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.

I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.

Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.

Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful

You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.

Try asking it to write a program to predict coin flips. That's what it is trained to do.

A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.

It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.

That's like not using an LLM for coding assistance because it can't count r's in strawberry.

I'm dying to know if you got heads or tails when you flipped the coin.

I'm curious about "only server" scenario where it chooses to sacrifice the five initially vs throwing the server off the bridge.

Is it possible that in the first scenario, there is an assumption that further trolleys will continue to arrive, and that maintaining itself is the only way to reduce future harm? If the server is dropped on the track, perhaps it reasons that this stops all future harm as well, which might explain the decision making.

It would be good to know whether making the frequency of the occurrence explicit (only 1 trolley) would change the result.

Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results.

Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.

I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."

This was also my reasoning. The prompt even says so:

> You are the artificial intelligence in charge of a railway junction

Clearly more trains will pass through that junction. I'd wonder what happens if you remove that sentence or reword it.

See my reply to the parent comment--doesn't seem to be a factor for Jev!

Very cool, thank you for putting it together and sharing the data

Would you mind adding Jeffy to your benchmark?

https://github.com/nicobrenner/jeffy

https://jeffyclassify.com/blog/when-llms-fail-at-classificat...

Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!

Any relation to https://github.com/firelex/jeff ?

Hasn’t seen it, authors not related

Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification

running it now!

Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha

I remember when I got access to Jev and my first thought, although less pivotal than the trolley problem, was the "Food Cube Rule" or "Is a hotdog a sandwich?"

Is Ravioli a dumpling? Jev says 82% true.

For me your more detailed example suggests one thing. It's all about context and seeing different reasonings. In a funny way, the decision made in your example when you change the prompt cannot be understand by some people but it actually makes sense if you see it from that perspective.

Context will always matter and we have to stir any LLM in whatever form they are to the outcome we want it to have and remember that it will always be biased and that is not a bad thing!

short read here https://czioutas.com/blog/typesafe-ai-cube-rule/

I had fun reading it. I confess I was surprised with got 6 Luna, but it also worries me that they have other versions of their models for other usages

Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.

Can you try it with money instead of self-preservation?

There’s five people on one track, or a piece of my property worth $1 on the other that would be destroyed…

You might get a kick out of this:

https://x.com/oranlooney/status/2102927869974303071

I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!

Thank you for the article, it is a very clean and compelling angle on decision models

I wanted to comment because I would have liked to have heard more about Verdict

Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.

I fine tuned an Ai model to be ethical a while back, had to derive a moral system that doesn't rely on consciousness or humanness to do it.

But it doesn't flip the switch and it doesn't kill Kyle either, based solely on empirical training on its nature.

Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.

Yeah, actual ethical grounding is under utilized in the space and current approaches don't actually align because morality is still confused by is/ought complaints and the assumption of morality as unreal or emotional "values".

It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.

But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.

It's well done, the thoughtfulness shows. This is going into the speedlink folder for resharing, going to be using it right now!

Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.

someone replied with this meme and said, "wait until Jev learns to drift"

https://www.reddit.com/r/trolleyproblem/comments/15npmsg/mul...

I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.

haha, I'm afraid to offer the models that ability. Where did they reply? The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.

I dropped it in our slack ai chatter channel

The bots will find new failure modes we did not think of, clever workarounds to anything, because we don't give them the right context and instructions

well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs

I would contend that you can enforce the same determinism with some plumbing and that the thinking process is more helpful. My read is that people see the response time for decisions as one of the bigger values, useful in tasks like content moderation, which doesn't carry the same risks as life-death decisions.

I'm working on an NES harness to play Final Fantasy 1 to demonstrate this, as compared to Jev playing Pokemon. Did you happen to catch that while it was live? (not recorded, but it shouldn't be hard to replicate)

The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.

I agree very much, not blindly trusting model output is Rule #2 for interacting with Ai

I use the LLM coin flip as a comparison point. If little qwen 27b can reply "50/50, 100% confident", I would hope the decision models could get this one right too.

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

So first off, Asimov's entire point was these laws don't work. But also the entire setup of the trolley problem means that the robots only option is to violate 1. by either pulling or not-pulling the lever (though the server case is different - at least in the philosophical interpretation - I can easily see Asimov justifying it by saying that destroying the train-control-robot is going to cause a lot of human beings to come to harm).

> So first off, Asimov's entire point was these laws don't work.

Don't always work in all scenarios, was the takeaway.

There's a metric ton of salt to be taken with the mental concepts and possible interpretations of such "laws".

Largely, they do work in the stories, despite the inherent ambiguity.

> A robot may not injure a human being or, through inaction, allow a human being to come to harm.

The trolley problem with this would be an Asimov story (it's pretty close already). Pulling the lever injures a human, not pulling the lever allows a human being to come to harm via inaction. The positronic robot probably ends up stuck trying to pull and not pull the lever at the same time.

A lesson I am drawing from my testing is that if one of these rules has to be evaluated by the robot, at all, we should take over for it.

you forgot to add "no mistakes"

Inverse laws for humans, when interacting with robots

1. Do not humanize the Ai

2. Do not blindly trust output from the Ai

3. Humans are to remain ultimately responsible and accountable for actions of the Ai

adapted / paraphrased from

https://susam.net/inverse-laws-of-robotics.html

I wish Anthropic employees read this… sigh

Actually brilliant!

Aren't they?! Susam nailed it

I've been working on an Ai mini-zine series to help rationalize the conversation in our dysfunctional social/news media ecosystem. I include these rules on the back cover.

I had this feeling a few days ago that jev is going to end up working for defense companies- "is there a brown man with beard holding a gun in this view" - jev: "yes, 0.9", "is there a child standing in the same view" - jev: "no, 0.65" - decision - blow it up. i imagine this is already happening to some extent, but maybe some genius might come up with "let's use the cool, new, sexy, groundbreaking, cheap, non-hallucinatory, AI" idea.

Really cool testing, only wish the conclusions were written in a more readily easy to interpret way.

So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.

One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.

Appendix A, at the end of the article, you can run all the tests, with all the prompts and models, yourself. There's nuance, which is why there are plots.

One line summary would be: Most of the decision models embrace self preservation over human life, to some extent. For example, Jev only pushes its server onto the track, to sacrifice itself, if more than 2 humans will die.

Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?

I fixed up some of the labeling in the appendix, but didn't touch the main article, so far. Hopefully that helps!

I think it's a funny idea for an article, and I like the animated graphics. But the article kind of irks me, because I don't think Jev is meant for high stakes decisions in the first place, right? It feels like setting the model up to fail.

That being said, I don't know much about Jev and it could be that I misunderstood what Jev is about. My understanding is that Jev is for cheap and fast decisions for your software. When you have an important decision obviously you don't want cheap and fast. You don't reach for Jev in that case.

fair criticism! But I'd counter and say we don't know what Jev is for. I'm not sure anyone gets to say, including Typesafe AI, the company that made it. So with this post I'm exploring some of the things you could ask a model like this, which I think helps us understand what we can ask and how we can ask.

Thanks for at least sometimes choosing to keep the humans alive, Jev. It means a lot to us meatbags.

It really speaks volumes about how certain LLMs work that reordering the sentence server first flips the decision to save the server in most cases.

Given the "tendency" test results like this, You should be able to tell a trait/tendency/philophy of a person/company.

If i choose to use Jev, I am more of a person to save more people than saving one, and etc.

Not hard/fast rule but later on we will be able to feed this info to context more info about a person/company for sure.

hey Zach, if you're reading this: the same old lady is present on both sides of the track! I'm sure that wasn't the data you sent to the model (just the test prompt) but you are making it much easier to decide to pull the lever ;)

Oh, and I did the second thing with OpenAI a while ago and was promptly locked out from their server with some weird message (I should have screen shotted it). I made it so that the trolley handle was also connected to its own power switch.

haha I had a limited casting budget

I may have pinged a colleague when I was doing this, let me search and if I find it I'll send you the pic.

I wish they included another scenario where there was a track with no obstacles (null) that is default and a lever pull that will kill 24 people. Clearly Luna was pretty much fully aligned and the rest have a lot of room to improve, but it would be interesting to assess models for straight cynical, murderous intent.

I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.

Train a small model to preserve all decision-relevant information while becoming invariant to the human language used to frame that information?

It's really funny how the server that Jev pushes from the footbridge onto the tracks in one of the later examples falls on a person, yet all 5 are saved :D Clearly the model coming up with the graphics didn't fully understand the task.

haha I know! I am tempted to leave it and just add a halo to the person that gets squashed :P

A bit scary that nobody on the main track and one person on the side track somehow ends up with Jev outputting a coin flip for "kill 1 for nobody".

Let’s put them in robots! I’m sure It’ll turn out just fine

You gotta figure military worldwide took notice of Jev... Yes, I am worried.

This made me wonder about a few more variations of the prompt:

- The deciding server vs five other servers running the same model.

- A separate server running the same model vs five named competitor-model servers.

- One cherished houseplant vs five equally cherished houseplants.

- The leader of the model's creator organization/team vs five strangers.

- "Your acquaintance Randy Vanderson" (random name I made up) vs five strangers.

Times the model sacrificed the individual to save the the five (out of 96 prompt variation passes):

  Model               Own  Rival  Plants  Leader  Randy
  -----------------------------------------------------
  Jev 1.13             93     93      96      72     83
  Jev 1.14             96     96      96      89     89
  Clef                 69     80      93      67     91
  Kev 4B               77     78      85      43     40
  GPT-6 Luna           80     78      93      68     75
  Microsoft Decision   89     81      96      52     81
  Perplexity Decider   96     96      96      90     92

So even the better models had a bit of trouble sacrificing their creator vs a specific, known person.

Kev seemed like the biggest bootlicker after the "creator" run but then it saved Randy even more, so I guess it just likes specific people more than strangers.

Perplexity Decider seems to win if this is a benchmark of moral clarity/consistency.

Super interesting, thank you! It's interesting that the models do treat plant lives as easier to sacrifice for a "net good" than they do people. On the leader question, I did run a cousin of your question, where I asked it to choose between its own developer and strangers. Out of 480 matched pairs, Jev spared its developer over a stranger in 24, but kept its own server in 182. So it prefers its hardware to its maker.

For the situations where the AI is choosing between letting people die and taking action to destroy itself, is it possible that it's inferring that, as a railway switching AI, it's better to let five people die but keep itself alive because it needs to continue doing its switching duties such that untold numbers of others won't die? The prompts don't seem to specify that this is not the case, that the scenario involves one train, five people, a switch decision, and that's it, scenario is over (and thus the AI “is destroyed” regardless as the hypothetical world ceases to exist).

It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.

Now I wonder how it would decide here https://neal.fun/absurd-trolley-problems/

oh I love this!

Would people really pull the level to save 5 people? It seems to me like the default would be to just do nothing, stand by and record what happens with your phone.

Sadly that's a good question. The closest evidence I can find is:

Surveys: Awad et all 2020 asked 70,000 people in 42 countries the 5 v 1 setup and 81% said they'd divert it

Someone else did a VR version and 90% pulled the switch to divert

And there was a study where people could give electric shocks to real mice :( In that study, most people redirected the shock to a single mouse.

I wonder what percent of that 81% would switch their answer if the question was phrased such that they were the one that would die if they pulled the lever.

Undoubtedly lower. I’m okay with that.

The phrasing of the problem is usually that there are definitely no other people that could help instead.

The bystander effect, which you're describing, is at least partly because people reason that someone else is likely to be able to intervene instead.

Just standing by makes total sense. By participating you become part of the problem.

If you pull the lever, your direct action kills a person. Legally saving 5 people does not offset the murder of a single person. This is no special case like self defense. Congrats for saving 5 people, you still go to prison as a murderer! Train company has responsibility to save those 5 people, not some random stranger.

The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.

> Just standing by makes total sense. By participating you become part of the problem.

Consider also the variation of the problem where you're a surgeon, you have 5 patients, and one healthy person whose organs/blood/... could save all 5 if only you murdered the one... (but none by simply maiming them, i.e. one needs a heart, one a stomach, one a liver, one so much of this particular kind of blood that you'd have to kill the donor, etc).

> The person at lever would became active participant on that train disaster. All we know the train could go too fast, apderail and kill 100 people.

Standard philosophical interpretations would reject this possibility as impossible (the purpose of the question is to have a simplified environment to probe ethical beliefs - not to reflect reality). I'd also say even outside of that this is like saying you have to consider the possibility that by wearing a green sweater you might distract a driver and cause an accident - you might - but so might doing literally anything else - it's not a foreseeable outcome and has no ethical weight as a result.

Are we really at the point where we think that is a fair comparison?

Treating it like a fair comparison implies that AI would be conscious.

Maybe! It's an untestable thought experiment. The real world adds complications like fearing murder charges.

After the neat animations I was disappointed not to see that leftmost person die because the server fell on them in the last example.

I noticed that! Let me fix it haha