Executive summary
AI systems are increasingly embedded in everyday life – but not always safely. While headlines focus on viral deepfakes, robotaxi crashes and chatbot controversies, a string of lesser-known incidents is revealing how AI can quietly cause harm at scale.
- A facial recognition system sent an innocent man to jail for two weeks.
- An algorithm stripped funding from 225,000 at-risk students across Nevada.
- A delivery robot injured a pedestrian on a college campus, then tried to run her over again.
- ChatGPT misrepresented key evidence in a child protection case, potentially endangering a vulnerable child.
- Medical transcription AI invented violent scenarios that never happened.
- An AI journalist falsely accused a district attorney of murder.
These AI failures share common threads: overreliance on automated systems, insufficient human oversight, and the dangerous assumption that sophisticated algorithms are infallible. Understanding these patterns can help organisations deploy AI safely while avoiding costly mistakes that damage both people and reputations.
Take a look around: we’re living through the largest deployment of AI systems in human history. Every day, algorithms make decisions about school funding, help police identify suspects, transcribe medical conversations, and navigate robots through public spaces. Most of the time, these systems work as intended. But when they don’t, the consequences can be dire. Real people lose jobs, face false accusations, get injured, or see their children’s educational opportunities vanish overnight.
The six incidents we’ll explore in this article aren’t the big, headline-grabbing AI disasters you’ve heard about (such as a chatbot wiping out US$100bn in market value with one wrong answer). Instead, they’re smaller-scale failures that nonetheless carry profound lessons for anyone deploying AI in consequential settings. Each reveals different ways in which machine learning systems can go horrendously wrong, and more importantly, how human oversight – or the lack thereof – and system design choices can either prevent or amplify these failures.
Wrongfully accused by AI
An innocent man spent 14 days in jail, simply because AI wrongly identified him as being involved in an armed robbery 1,500 miles away.
Facial recognition technology has become ubiquitous in law enforcement in recent years, with police departments worldwide increasingly using AI-powered systems to identify suspects from grainy surveillance footage. The appeal is obvious: faster investigations, reduced manual labour, and the promise of objective, unbiased identification. Many police departments report that facial recognition has helped them solve cold cases and identify suspects who might have otherwise escaped detection. The technology isn’t limited to law enforcement applications. Major retailers also use similar systems to flag known shoplifters, while airports employ the technology to enhance security screening.
The market for facial recognition systems reached billions of dollars annually, driven by perceived accuracy rates that often exceed 95% in controlled laboratory settings. However, as many people would later find out, laboratory performance rarely translates perfectly to real-world deployment. Facial recognition systems often struggle with poor lighting, unusual angles, low-resolution images, and the inherent variability in human appearance. But what’s even more concerning is how organisations implement these tools: many treat AI-generated matches as definitive evidence rather than probabilistic assessments that require careful verification.
Harvey Murphy Jr. experienced the most devastating possible outcome of facial recognition failure when a system used by a retail chain identified him as an armed robber in Houston – a city he had never even visited. The AI flagged Murphy’s photo as a potential match for surveillance footage from the crime, and investigators treated it as sufficient evidence to issue an arrest warrant. Police arrested Murphy during a routine DMV visit, and he spent nearly two weeks in jail protesting his innocence. During his detention, Murphy was repeatedly assaulted by other inmates, suffering permanent injuries that will affect him for the rest of his life.
Only after prosecutors confirmed he was actually in Sacramento at the time of the Houston robbery was he released. Murphy is now suing the companies involved for $10 million, but no amount of money can undo the trauma he experienced because humans trusted a machine’s flawed judgment. Sadly, this wasn’t a one-off glitch. A subsequent Washington Post investigation found that at least eight Americans have been wrongfully arrested due to facial recognition errors, with many cases involving police who ignored contradictory evidence like airtight alibis or clear physical differences between suspects and the arrested individuals.
Students at risk
An AI meant to identify struggling students decided poor kids with good grades didn’t need help, stripping 225,000 of their funding.
Educational funding decisions affect millions of children, making accuracy and fairness paramount. Traditionally, these decisions relied on straightforward criteria like family income levels, ensuring that resources reached students facing economic disadvantages. The main problem with this approach was that determinations were made almost exclusively based on a student’s financial situation, ignoring many other factors that could have affected the student’s educational outcomes. Nevada’s education department saw an opportunity to improve this system by using machine learning to identify students at the highest risk of not graduating.
The goal was admirable: target limited resources more precisely to help those who needed support most. To this end, the state partnered with Infinite Campus, an educational technology firm, to develop a predictive model that would analyse 75 different factors – ranging from grades and attendance to behavioural incidents – and generate a ‘graduation score’ for each student, similar to credit scores but predicting educational outcomes rather than financial behaviour. Only students scoring below a certain threshold would be classified as ‘at-risk’ and eligible for additional funding and support services.
Unfortunately, the decision to delegate this task to AI backfired in spectacular fashion. Previously, around 288,000 students (roughly 60% of Nevada’s K-12 population) qualified for ‘at-risk’ funding based largely on family income. The new AI system reduced this number to just 63,000 students, only 13% of the state’s student body. That means that nearly three-quarters of students who had been receiving additional support suddenly lost that status overnight. The funding cuts hit schools in low-income communities particularly hard. One charter network saw its at-risk student count plunge from 1,700 to just 45, forcing administrators to eliminate tutoring programs and other support services.
The algorithm’s narrow definition of risk meant that many students from poor families lost funding simply because they had decent grades and attendance. While their economic challenges hadn’t disappeared, the AI decided they weren’t risky enough to warrant the economic support. The lack of transparency in the algorithm’s decision-making process invariably made it impossible for stakeholders to understand or challenge specific determinations, further eroding trust in the system. This led to widespread backlash from educators and parents, who argued that the system had essentially redefined which students deserved help based on algorithmic judgments rather than actual need. The controversy ultimately forced Nevada officials to reconsider their approach, but not before thousands of students lost access to programs designed to help them succeed.
Robots on rampage
A delivery robot knocked down a university employee, then reversed and tried to run her over while she lay injured on the ground.
Autonomous delivery robots represent one of AI’s most visible applications in daily life. These cooler-sized machines stroll sidewalks and campus paths, using camera vision and machine learning to avoid obstacles while delivering food and packages. Companies like Starship Technologies have thus far deployed thousands of these robots across university campuses and urban areas – mainly in California and China – completing millions of deliveries with generally impressive safety records. The robots move slowly, typically at walking speed, and are equipped with multiple sensor systems to safely detect and avoid pedestrians, vehicles, and other obstacles.
The technology relies on real-time processing of visual and spatial data to make navigation decisions. Cameras, LiDAR sensors, and mapping systems work together to create a three-dimensional, sensor-redundant understanding of the robot’s environment. Machine learning algorithms then interpret this sensory data to plan safe paths and respond to dynamic situations like people walking nearby or vehicles approaching. However, like all AI systems, delivery robots can also struggle with unusual or ambiguous situations that fall outside the scope of their usual training data.
At Arizona State University in September 2024, a routine delivery went catastrophically wrong when a Starship robot collided headfirst with a campus employee. According to the police report, the robot had initially stopped to let the woman pass, demonstrating that its sensors had detected her presence. However, as she resumed walking, the robot suddenly reversed direction and knocked her to the ground, causing a lower back injury and a four-inch cut on her arm that required medical treatment. The incident became even more disturbing when the robot, after initially moving away from the fallen pedestrian, reversed again and drove toward her while she was still on the ground.
While at face value this hit-and-run (or perhaps more accurately, hit-and-hit-some-more) incident suggests the AI system failed to recognise that what it had struck was a person who needed assistance, instead classifying her as some kind of obstacle it could navigate around or over. Starship later explained that the robot was responding to an oncoming vehicle, but this raises questions about the system’s ability to prioritise different types of hazards appropriately. The company’s initial response – offering the injured woman promotional codes for free robot deliveries – was lambasted for its tone-deafness and highlighted how unprepared many autonomous mobility firms really are for handling real-world safety incidents.
ChatGPT endangers a child’s safety
ChatGPT turned key evidence against an abusive father into a glowing parenting review, nearly endangering a child’s safety.
Child protection work involves some of the highest-stakes decisions in government, with workers required to assess complex family situations and make recommendations that can determine whether children remain with their parents or enter foster care. The workload is often crushing, with case workers juggling dozens of active cases simultaneously and facing tight deadlines for court reports and assessments. Given these pressures, the appeal of AI writing assistance is easy to see: tools like ChatGPT can help draft reports, summarise case notes, and handle routine documentation tasks.
However, large language models like ChatGPT are fundamentally designed to generate plausible-sounding text, not to ensure factual accuracy. They work by predicting the most likely next word based on patterns learned from training data, which means they can confidently produce information that sounds authoritative but is completely fabricated. Imagine the consequences, then, when a child protection worker in Victoria, Australia, decided to use ChatGPT to help draft a report for family court proceedings. The AI-generated content included a serious factual error that completely mischaracterised crucial evidence in the case. The report described a child’s doll, which had allegedly been used by the father for disturbing purposes, as simply an example of an ‘age-appropriate toy’ that demonstrated the father’s commitment to supporting the child’s development.
This wasn’t a simple typo or misunderstanding. The doll was key evidence in the case against the father, and by portraying it as a positive parenting choice, the AI had essentially flipped the narrative from concerning to supportive. If the error hadn’t been caught during review, it could have led a judge to gravely underestimate the risk to the child and potentially grant the father more access or custody rights. The incident also violated privacy protocols, as the worker had entered confidential case details into an external AI system. Following an investigation that highlighted these risks, Victoria’s child protection department banned staff from using generative AI tools for any casework.
AI’s violent fantasies
OpenAI’s medical transcription service invented violent fantasies and inserted them into patient records.
Healthcare documentation consumes enormous amounts of physician time, with doctors spending nearly two hours on administrative tasks for every hour of patient care. AI transcription services promise to solve this problem by automatically converting patient conversations into accurate medical notes, freeing healthcare providers to focus on patient care rather than paperwork. OpenAI’s Whisper, one of the most advanced speech recognition models currently available, has thus far been integrated into healthcare applications serving over 30,000 medical workers across 40 health systems. The system promises near-human accuracy in converting speech to text, with particular strength in handling medical terminology and complex conversations.
Whisper represents a significant advance over traditional services, using deep learning techniques trained on 680,000 hours of audio data to produce transcriptions at inhuman speeds. Rather than simply matching sounds to words, the system attempts to understand context and generate coherent text that reflects the speaker’s intent. However, researchers discovered a troubling characteristic: when faced with unclear audio, background noise, or ambiguous speech, Whisper doesn’t simply mark sections as inaudible or provide phonetic guesses. Instead, it generates complete, confident-sounding sentences that sometimes turn out to be utter nonsense. It doesn’t take a genius to see the danger of this in a medical context.
In one documented case, innocuous speech about a boy and an umbrella was transcribed as luridly violent content about killing people with a knife. The AI had completely invented a violent scenario that bore no resemblance to the original conversation whatsoever. Another test revealed racial bias in the system’s hallucinations: when a speaker mentioned “two other girls and one lady,” Whisper added the fabricated detail that “they were Black,” inserting racial information that was never spoken.
University of Michigan researchers found that Whisper added invented content to 80% of the public meeting transcripts they analysed, while another developer reported hallucinations in nearly all of 26,000 test audio files. The healthcare implications are particularly concerning because some implementations delete the original audio after transcription for ‘privacy’ reasons, making it effectively impossible to verify accuracy later. Doctors reviewing these transcripts have no way to distinguish real patient statements from AI hallucinations, potentially leading to dangerous misdiagnoses or inappropriate treatments based on fabricated information.
A murder mystery mix-up
An AI journalist falsely accused a district attorney of murder based on his own office’s social media post.
Journalism faces unprecedented economic pressures, with newsrooms cutting staff while trying to maintain coverage of community events, government meetings, and public safety incidents. AI-generated content offers an appealing solution: algorithms can monitor social media feeds, police reports, and official announcements, then automatically generate news articles about local events. This approach promises to restore consistent local news coverage at a fraction of traditional costs. However, large language models used for automated journalism face the same fundamental limitations as other AI text generators. They excel at producing fluent, grammatically correct prose, but they lack a genuine understanding of context, causation, or factual relationships.
When these systems process official communications or social media posts, they must infer meaning from limited textual cues, often filling gaps with statistically probable but potentially incorrect information. In October 2024, readers of Hoodline San Francisco encountered a shocking headline claiming that the San Mateo County District Attorney had been charged with murder. The AI writing system had processed a social media post from the DA’s office announcing an arrest in a murder case, but fundamentally misunderstood the information. Instead of recognising that the post came from the DA’s office about someone else’s arrest, the AI merged the account name San Mateo County DA with details about criminal charges, creating a narrative where the DA himself was the perpetrator.
The system even invented a fictional name for the supposedly arrested DA, John Thompson, combining text fragments in a way that sounded plausible but was completely fabricated. The article included specific details about a preliminary hearing and the alleged victim, creating a comprehensive but entirely false news story. A tiny AI label appeared next to the byline, but unfortunately, no human editor had reviewed the content prior to publication. The false story was indexed by Google News and served to users searching for local information, amplifying the misinformation before Hoodline caught and corrected it.
Learnings
These six incidents share a common thread: AI systems failing in ways their creators didn’t anticipate, causing real harm to real people. From wrongful arrests and lost education funding to fabricated medical records and false murder accusations, we’re seeing what happens when witless AI systems meet the chaotic complexity of human life. The near future will likely bring stricter regulations as governments react to these failures. Expect mandatory human oversight requirements for AI in high-stakes domains, transparency rules for algorithmic decision-making, and liability frameworks that hold companies accountable for their AI’s mistakes.
Forward-thinking organisations can’t afford to sit around and wait for regulation – they’ll implement robust testing, clear accountability, and human-in-the-loop safeguards now. Looking further ahead, the challenge isn’t making AI perfect, but making it safe enough to trust with important decisions. That means accepting that failures will happen and building systems to catch them before they cascade into disasters. The question for every business leader isn’t whether your AI will fail, but whether you’ll be ready when it does. How will you know when your AI is confidently wrong? And more importantly, what will you do about it?
