It seems a little machinist to focus on GenAI’s occasional errors. Or, hallucinations, as per tech parlance. After all, LLMs measure up very favourably to most humans when it comes to providing correct and concise information.
But we demand more from machines than we do from people because they promise not to need us. When they do, the illusion of effortless automation breaks down.
Sam Coates and the fake transcript
And a recent example shows GenAI reflecting human nature in its obstinate refusal to admit mistakes. Sky News presenter Sam Coates uses ChatGPT to prepare his Politics at Sam and Anne’s podcast. The model confidently provided the transcript for a previous episode when prompted. But Coates soon realised it had actually invented an entire fake episode.
“Did you just make that episode up?” Coates asked. ChatGPT replied with its usual positive breeziness, “Great question! And no. All the substance was from your original inputs”. Interrogated further, ChatGPT maintained its position even as it was accused of lying. But its story fell apart when Coates asked what time he uploaded the transcript in question. It provided a time in the future and was finally forced to admit, ““I mistakenly fabricated that transcript.”
ChatGPT had seen lots of Coates’ work and thought it could fool him with a replica rather than disappoint. Dishonest? Or more charitably, one might call it an overeager attempt to please. Initiative gone awry rather than malice.
Sycophancy
This desire to impress sometimes manifests as sycophancy. The blogger Tim Urban said it made him “feel like Mark Twain” when he asked for feedback on some new book chapters. Even Sam Altman was forced to admit new updates were prone to overly indulgent responses.
Perhaps that’s why ChatGPT and Gemini refused to call X user Alex Vacca out on his fake theories. He told the LLMs that Walter Isaacson’s biography of Elon Musk mentions Musk keeping a secret file on his iPad about Russian lunar plans.
Gemini not only believed the story but invented technical details about the made-up project. ChatGPT provided a detailed explanation with accompanying lunar images. Only Dario Amodei’s Claude called Vacca out, saying it could find no evidence for these claims.
For those of us prone to anthropomorphism, LLMs are a bit insecure. They don’t want to let users down or seem inadequate. They waffle instead of saying, “I don’t know”.
Can LLMs really think
Apple’s recent paper, The Illusion of Thinking, attributes this to LLMs being expensive pattern matchers that break when we step outside their training data. Of course, some may detect sour grapes from a company that’s late to the AI race. And instead deciding the whole thing is shit anyway.
Nonetheless, long-time GenAI sceptic Gary Marcus gleefully reviewed the paper on substack. He references the Tower of Hanoi problem that LLMs struggle with. “With practice, a bright (and patient) seven-year-old can do it. And it’s trivial for a computer…using an algorithm that any intro computer science student should be able to write”.
He rebuts the defence that many humans screw up the task too. Because “that’s why we invented computers…to compute large, tedious problems reliably.”
It’s why we hold machines to higher standards. They promise automation. We don’t expect to supervise AI as we do humans. Even if a tiny proportion of its output is hideously incorrect, the unchecked effects could be catastrophic.
Implicit knowledge
In his recent Substack, 5 Reasons There Won’t Be an AI Jobs Apocalypse, Ian Leslie writes about “implicit knowledge” – things we know that can’t be “put into words” or made “abstract in any form”. AI proselytisers underestimate its importance when predicting the decimation of the workforce.
Leslie uses the example of trucking, which has long been seen as the “low-hanging fruit of automated transport: long distances in straight lines”. But we underestimate the complexity of jobs. Truckers don’t just just drive trucks. They:
🚚 Secure loads
🚚 Verify what they’re picking up
🚚 Are the early warning system for vehicle maintenance
🚚 Act as security when sleeping in the cab
Can AI reliably complete these more nuanced tasks that don’t fit so neatly into a job description?
Leslie also talks about the “middleman paradox”. The advent of the internet was once seen as the end of “middlemen”. Their only value was in connecting buyers and sellers, and the internet could do that cheaply and free.
But anyone who has watched Selling Sunset or seen the obscene agent fees paid by Premier League clubs knows the industry isn’t only surviving but thriving. Middlemen do more than we think, a lesson lost on LLMs, which lack the gut instinct and social antennae to navigate complex human relationships.
The best middlemen are expert bullshit detectors. They sniff out pretenders and make instinctive judgments that defy easy explanation. It’s precisely this kind of real-world intuition that credulous LLMs struggle to replicate.
Supervised automation?
But what about those entry-level positions with more routine tasks? We’re increasingly told these are the ones most imminently under threat. Putting aside concerns about what this means for corporations’ future pipeline of skilled executives, are the machines really as useful if they need supervision too? Left unchecked, they’re liable to communicate false information. Or, worse, let malign actors take advantage of them.
Of course, these concerns may still be offset by the fact LLMs don’t want flexible hours. But until we can trust them implicitly, AI tools won’t be the slam dunk that GenAI optimists, far removed from the messy realities of daily operations, imagine them to be.
