HN.zip

We asked AI the same 210 buying questions 2 days in a row,84% of answers changed

17 points by - 4 comments
[3 hidden]5 mins ago
I don't think this is inherently surprising.

Looking at the questions they asked, they are quite broad. In my experience, part of the value of LLMs comes from its randomness, but you must understand it as a characteristic virtue of these tools, and must adapt accordingly.

Asking questions with tight scopes, clear constraints, and specific outcomes, always yields better results, in my opinion.

[3 hidden]5 mins ago
In general it seems like LLMs have a fundamental problemw with noise in their outputs such that randomness (what data they pull and how they aggregate it) has an outsized impact when asking it for opinions/recommendations/forecasts.
[3 hidden]5 mins ago
I’m the author. Every day we ask ChatGPT, Gemini, Google AI Overviews and Google AI Mode the same 210 buying questions, like “best CRM for small business”, and record which brands they name and which pages they cite. This post compares the first two days.

84% of answers named a different set of brands the next day. On Gemini it was 95%.

The cause is easy to see in the data. Each answer is a summary of a few web pages, and the engine picks those pages fresh each time. Fewer than half of the pages cited on day one were cited again on day two. When the pages change, the brands change.

One example: asked for the “easiest CRM for a team that has never used one”, Google AI Mode read Zapier’s list and put Less Annoying CRM first. The next day it read a Slack blog post instead, and Less Annoying CRM wasn’t mentioned at all.

Who survives depends on how many pages you’re on. A brand ranked #1 was named again the next day 90% of the time. A brand ranked sixth or lower: 61%. Big brands are on most of the pages, so whichever ones the engine reads, they’re there. Smaller brands are on a few, so they come and go.

[3 hidden]5 mins ago
??? Set the temperature to zero then ???