Interesting to note how similar this seems to what happened with Benj Edwards at Ars Technica. AI was used to extract or summarize information, and quotes found in the summary were then used as source material for the final writing and never double checked against the actual source.
I’ve run into a similar problem myself - working with a big transcript, I asked an AI to pull out passages that related to a certain topic, and only because of oddities in the timestamps extracted did I realize that most of the quotes did not exist in the source at all.
crop_rotation [3 hidden]5 mins ago
HN is full of people saying ABCD should know better and honestly I thought the same, but when I look at almost all of my friends working in critical domains like as a judge or engineer or lawyer or even doctor, they seem to trust ChatGPT more or less blindly. People get defensive when I point out out to them that ChatGPT will make things up and it is widely know, and some even tell me it is the fault of "tech people" for not fixing it and they can't be expected to double check every chatgpt conversation. So I am very sure this problem is more prevalent than what we see and also that it is going to continue increasing.
andrewflnr [3 hidden]5 mins ago
Your friends should know better. That their behavior is prevalent does not contradict that.
crop_rotation [3 hidden]5 mins ago
Yes and the world should be utopia and everyone should be happy and we all wish for world peace and yada yada yada. What you are saying is a vision of ideal world as it should be, but doesn't help anyone understand the real world problems.
andrewflnr [3 hidden]5 mins ago
You can't seriously compare the problem of world peace with the problem of exercising the most basic level of critical thinking w.r.t. LLM output after it has already proven itself unreliable. That's not a utopian dream, it's a level of prudence on par with not sticking a fork in an electrical socket.
joe_mamba [3 hidden]5 mins ago
>but when I look at almost all of my friends working in critical domains like as a judge or engineer or lawyer or even doctor, they seem to trust ChatGPT more or less blindly
That's why I lost trust and faith in people who end up in positions of doctor, lawyer or judge. When I was young I used to think they must be the smartest most high-IQ people in society, having read the most books and have the highest levels of critical thinking and debate skills ever. When in fact they were only good at memorizing and regurgitating the right information that the school required to pass the exam that gave them that prestigious title and that's it.
Now in my mid 30's when I talk to people in these professions at a beer or any other casual gathering, I realize they're really not that sharp or well read, and anyone could be in their place if they put in the grind work at the right time. It's a miracle our society functions at all.
doctorpangloss [3 hidden]5 mins ago
on the flip side, so much chatgpt usage, full of flaws, doesn't seem to really matter in various "critical domains." you can't generalize "critical."
Chinjut [3 hidden]5 mins ago
Good lord, even the apology is AI generated: "That was not just careless—it was wrong."
His non-apology apology even follows a familiar pattern: I wrote it myself but just used AI for some help, and it inserted false quotes! Bad tech! But I have now learned my lesson!
Very similar to what a rector recently wrote when she got busted giving an AI-generated speech in her inaugural speech in her new university job.
None of it is true, of course. These people are just sorry they got caught.
hvb2 [3 hidden]5 mins ago
I think his apology was actually written in Dutch so this might be a translation that was automated?
It is a faithful translation of the original Dutch. Dutch is structurally very similar to English so this type of nuance carries over pretty much intact.
Dutch: “Dat was niet enkel onzorgvuldig, het was fout.”
English: “That was not just careless—it was wrong.”
I’d say the only difference is the em dash.
Whether you consider it proof of AI is up to y’all.
rsynnott [3 hidden]5 mins ago
Particularly given that the dreaded em-dash is not commonly used in Irish or UK English; it’s mostly a US English thing.
microtonal [3 hidden]5 mins ago
The original (?) apology in Dutch does not use em-dashes:
I’m tempted to agree, but this is a case where I think there’s more human than AI. Maybe he used LLMs for a bit, and changed parts of it. Maybe he is patient zero for LLM speak?
ashwinnair99 [3 hidden]5 mins ago
The tool didn't fail here, the person did. An experienced journalist should know better. Editorial review exists for exactly this reason, if you skip it, this is what happens.
microtonal [3 hidden]5 mins ago
But the article said he published it in his own Substack newsletter, I am assuming that it is not under editorial control, since it is personal?
Hendrikto [3 hidden]5 mins ago
> The tool didn't fail here, the person did
Both failed.
camillomiller [3 hidden]5 mins ago
I have witnessed in person what LLMs have done to the mind of seemingly intelligent people. It’s a disaster.
cinntaile [3 hidden]5 mins ago
Don't leave us hanging. What happened?
camillomiller [3 hidden]5 mins ago
A CTO sent me a message that opened with:
“Here’s a friendly message that will perfectly convey what you want to say”.
A double PhD friend says she has to talk to chatGPT for all sort of advice and can’t feel safe not doing it, “because you know I’m single and don’t have a companion to spitball my ideas”. She let chatGPT decide which way to take to get to a certain island, and she got stranded because the suggested service didn’t exist.
I have more examples. It’s a fucking mind virus.
sigseg1v [3 hidden]5 mins ago
How is the getting stranded example different than asking on a travel forum how to get somewhere, and an active and well intentioned user that isn't familiar with your area of travel answers, gives you wrong instructions, and you get lost?
andrewflnr [3 hidden]5 mins ago
The key missing step is where the traveler exercises critical thinking and checks the advice they get. Some people seem to turn that off for LLMs.
shahbaby [3 hidden]5 mins ago
Because they aren't probabilistic parrots? If they get it wrong, there's usually an understandable reason behind it.
kibwen [3 hidden]5 mins ago
Because the vast and overwhelmingly majority of the time, if you ask a question into the ether that nobody has a good answer to, most people will gloss over it and not bother answering, as attested by decades of relatable memes ( https://xkcd.com/979/ ). In contrast, the chatbot is trained to always attempt to give an answer, and is seemingly disincentivized via its training set to just shrug and say "I don't know, good luck fam".
dude250711 [3 hidden]5 mins ago
They stop thinking and they stop verifying output too.
maxrmk [3 hidden]5 mins ago
Ironic coming from the Guardian. One of their journalists consistently publishes ai slop and the paper is in denial about it.
It doesn't seem AI generated to me. Are we at the point where you have to write in a particularly outrageous style in order to not be accused of using AI?
gruez [3 hidden]5 mins ago
>Are we at the point where you have to write in a particularly outrageous style in order to not be accused of using AI?
I don't think we've gotten to the extent that all popular writing styles (eg. hamburger paragraphs) are considered suspect, but the "it's not just X, it's Y" construction[1] attracts particular scrutiny.
I was giving this the benefit of the doubt as well and was just looking at his older writings that have a little "This article is more than 5 years old" banner above it. Looks totally different indeed.
maxrmk [3 hidden]5 mins ago
Fair enough. It reads as extremely AI generated to me. But that isn’t completely reliable.
shahbaby [3 hidden]5 mins ago
> That was not just careless – it was wrong
lol
PeterStuer [3 hidden]5 mins ago
"Journalism" over here seems to have died a long time ago. Most if not all of the former "quality newspapers" unfortunately seem to have devolved into what could be more accurately described as "pro regime activist blogs".
intended [3 hidden]5 mins ago
Looking at the media ecosystem at large, gives me a case of gallows humor.
In some sections of the ecosystem, firms still penalize journalists for errors. In other sections, checking reduces the velocity of attention grabbing headlines. The difference in treatment is… farcical.
We need more good journalists, and more good journalism - but we no longer have ways to subsidize such work. Ads / classifieds are dead, and revenue accrues to only a few.
I have no idea how we square this circle.
PeterStuer [3 hidden]5 mins ago
We can't square this circle. It's why they're all A/B flipping headlines (resulting in the most deranged partisan clickbait), killed of their (too expensive) redactions (especially international news), rely solely on (barely) rewriting AP, Reuters and PRNewswire, and fill their site with opinion rather than factual reporting in support of gov handouts to the sector.
mmooss [3 hidden]5 mins ago
They said earlier that they didn't verify the quotes. I understand them to mean that the LLM outputted text that included quotes. They assumed the output was accurate and found it so appealing, on an emotional level, that they just went with it without checking.
The most valuable lesson here, by far, is not about other people but about ourselves. This person is trained, takes it seriously, and advocates for making sure the AI is supervised, and got caught in the emotional manipulation of LLM design [0].
We all are at risk. If we look at the other person and mock them, and think we are better than them, we are only exposing ourselves to more risk. If we think - oh my goodness, look what happened, this is perilous - then we gain from what happened and can protect ourselves.
(We might also ask why this valuable tool also includes such manipulative interface. Don't take it for granted; it's not at all necessary for LLMs to work, and they could just as easily sound like a-holes.)
[0] I mean that obviously they are carefully designed to sound appealing
phreack [3 hidden]5 mins ago
> “It is particularly painful that I made precisely the mistake I have repeatedly warned colleagues about: these language models are so good that they produce irresistible quotes you are tempted to use as an author. Of course, I should have verified them. The necessary ‘human oversight’, which I consistently advocate, fell short.”
What? Irresistible quotes? This betrays a terrible way of thinking as a journalist. Basically an admission of wanting to fake news that'd sound good. At that point just write fiction.
Obscurity4340 [3 hidden]5 mins ago
Cant you, like, ask or instruct it to create a bibliography with the citations or at least put the source of any quotes next to it for reviewing purposes?
sofixa [3 hidden]5 mins ago
> Basically an admission of wanting to fake news that'd sound good
How did you read that? Something sounding good and making sense and you wanting it to be true doesn't mean you'd fake it.
abaieorro [3 hidden]5 mins ago
> I wrongly put words into people’s mouths, when I should have presented them as paraphrases
Journalists were doing this for decades. Stitching and editing words out of context, to put words into peoples mouths! I will take AI halucinations over journalists halucinations anytime, at least machine has no hostile intent, and is making a geunine error!
garciansmith [3 hidden]5 mins ago
The idea that somehow AI is magically unbiased and not influenced by those making it is incorrect.
hulitu [3 hidden]5 mins ago
> I will take AI halucinations over journalists halucinations anytime, at least machine has no hostile intent,
Famous last words. What do you think is the main application for AI ? Spreading propaganda.
HN.zip
I’ve run into a similar problem myself - working with a big transcript, I asked an AI to pull out passages that related to a certain topic, and only because of oddities in the timestamps extracted did I realize that most of the quotes did not exist in the source at all.
That's why I lost trust and faith in people who end up in positions of doctor, lawyer or judge. When I was young I used to think they must be the smartest most high-IQ people in society, having read the most books and have the highest levels of critical thinking and debate skills ever. When in fact they were only good at memorizing and regurgitating the right information that the school required to pass the exam that gave them that prestigious title and that's it.
Now in my mid 30's when I talk to people in these professions at a beer or any other casual gathering, I realize they're really not that sharp or well read, and anyone could be in their place if they put in the grind work at the right time. It's a miracle our society functions at all.
https://pressanddemocracy.substack.com/p/i-am-admitting-my-m...
Very similar to what a rector recently wrote when she got busted giving an AI-generated speech in her inaugural speech in her new university job.
None of it is true, of course. These people are just sorry they got caught.
Source: https://www.linkedin.com/posts/peter-vandermeersch-a4381b30_...
Dutch: “Dat was niet enkel onzorgvuldig, het was fout.”
English: “That was not just careless—it was wrong.”
I’d say the only difference is the em dash.
Whether you consider it proof of AI is up to y’all.
https://steady.page/en/journalistiekondervuur/posts/dd6e066f...
Both failed.
“Here’s a friendly message that will perfectly convey what you want to say”.
A double PhD friend says she has to talk to chatGPT for all sort of advice and can’t feel safe not doing it, “because you know I’m single and don’t have a companion to spitball my ideas”. She let chatGPT decide which way to take to get to a certain island, and she got stranded because the suggested service didn’t exist.
I have more examples. It’s a fucking mind virus.
https://x.com/maxwelltani/status/2023089526445371777?s=46
I don't think we've gotten to the extent that all popular writing styles (eg. hamburger paragraphs) are considered suspect, but the "it's not just X, it's Y" construction[1] attracts particular scrutiny.
[1] https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing#...
[1] https://xcancel.com/maxwelltani/status/2023089526445371777?
lol
In some sections of the ecosystem, firms still penalize journalists for errors. In other sections, checking reduces the velocity of attention grabbing headlines. The difference in treatment is… farcical.
We need more good journalists, and more good journalism - but we no longer have ways to subsidize such work. Ads / classifieds are dead, and revenue accrues to only a few.
I have no idea how we square this circle.
The most valuable lesson here, by far, is not about other people but about ourselves. This person is trained, takes it seriously, and advocates for making sure the AI is supervised, and got caught in the emotional manipulation of LLM design [0].
We all are at risk. If we look at the other person and mock them, and think we are better than them, we are only exposing ourselves to more risk. If we think - oh my goodness, look what happened, this is perilous - then we gain from what happened and can protect ourselves.
(We might also ask why this valuable tool also includes such manipulative interface. Don't take it for granted; it's not at all necessary for LLMs to work, and they could just as easily sound like a-holes.)
[0] I mean that obviously they are carefully designed to sound appealing
What? Irresistible quotes? This betrays a terrible way of thinking as a journalist. Basically an admission of wanting to fake news that'd sound good. At that point just write fiction.
How did you read that? Something sounding good and making sense and you wanting it to be true doesn't mean you'd fake it.
Journalists were doing this for decades. Stitching and editing words out of context, to put words into peoples mouths! I will take AI halucinations over journalists halucinations anytime, at least machine has no hostile intent, and is making a geunine error!
Famous last words. What do you think is the main application for AI ? Spreading propaganda.