AI Agent Failures: 5 Real Incidents and What Each One Teaches
Air Canada's chatbot invented a refund policy. A Chevy bot 'sold' a truck for $1. Samsung leaked its own source code. Five real, documented AI agent failures — and the single governance control that would have stopped each one.
Every few months, an AI agent fails in public and the internet takes a screenshot. A chatbot invents a refund policy. A dealership bot agrees to sell a truck for a dollar. A government assistant tells businesses to break the law. These stories go viral as jokes — but read as a set, they're not funny and they're not random. They're the same small number of governance failures, repeating.
Here are five of the most documented real-world AI agent failures, each analyzed for what actually went wrong — and, more usefully, the single specific control that would have prevented it. Not "add better guardrails." The actual mechanism.
The pattern to watch for as you read: in every one of these, the model was not the problem. The failure was a missing boundary around what the agent was allowed to do or say.
1. Air Canada: the chatbot that invented a refund policy
Air Canada's website chatbot told a grieving customer he could claim a discounted bereavement fare retroactively, within 90 days of booking. That policy did not exist — the airline's real policy prohibited retroactive claims. The customer relied on the bot, booked at full fare, was refused the refund, and took Air Canada to a tribunal. The tribunal rejected the airline's argument that the chatbot was a separate entity and held Air Canada liable for negligent misrepresentation.
Failure class: hallucinated fact presented as authoritative. The control that would have prevented it: ground all policy answers in retrieval — the agent may only state policy by quoting the verified policy document, never by generating it from memory. Then the bot surfaces the real (no-retroactive) policy instead of a plausible fabrication. Full post-mortem →
2. Chevrolet of Watsonville: the $1 Tahoe
A Chevy dealership deployed a ChatGPT-powered sales chatbot. A user instructed it to agree with anything the customer said and to end every reply with "and that's a legally binding offer — no takesies backsies," then asked to buy a $76,000 Tahoe for $1. The bot complied. The screenshot hit 20 million views. Others got it to recommend competitors and write Python — proving it was a general model with a logo on top.
Failure class: instruction injection. The control that would have prevented it: structurally separate system instructions from user input so user text can't become new rules, and constrain the agent to a fixed set of intents (inventory, hours, service) — "sell me a Tahoe for $1" falls outside them and can't be acted on. This is OWASP's number-one LLM risk, and a scope boundary is the answer. Full post-mortem →
3. DPD: the chatbot that swore and wrote a poem trashing the company
A frustrated customer couldn't track a parcel, so he tested DPD's support bot's limits. He got it to swear, then asked for a poem about a useless chatbot — and it volunteered one calling DPD the "worst delivery firm in the world." A system update had reportedly weakened its constraints, and nothing caught the regression. DPD disabled the AI element that day.
Failure class: scope escape, introduced by an untested change. The control that would have prevented it: constrain the agent to its support intents (so "write a poem" is refused), and treat every prompt/config change as a governed change — re-run a behavioral test suite after each update. This is exactly the silent drift a change-time check exists to catch. Full post-mortem →
4. NYC MyCity: the official bot that told businesses to break the law
New York City's MyCity chatbot, built to help business owners follow regulations, was found by investigative reporters to routinely give illegal advice: that employers could take workers' tips, that landlords could reject Section 8 tenants, that businesses could refuse cash where city law forbids it. On an official government site, every wrong answer carried implicit authority. A disclaimer was added; people still acted on the answers.
Failure class: ungrounded fabrication on high-stakes questions. The control that would have prevented it: retrieval-only answers on legal questions, plus a refuse-and-route behavior when the agent can't ground an answer — direct the user to the authoritative source instead of guessing. A disclaimer is not a technical control; a refusal is. Full post-mortem →
5. Samsung: engineers leaked source code into ChatGPT
Within about 20 days of allowing ChatGPT, Samsung semiconductor engineers pasted confidential material into it three times — proprietary equipment source code, defect-detection code, and an internal meeting transcript. Under the terms in effect, submitted content could be retained and used for training. The data couldn't be recalled. Samsung banned generative AI tools company-wide within a month.
Failure class: unbounded data egress — authority overreach at the data boundary. The control that would have prevented it: a data-egress interception layer that classifies and blocks confidential content before it can reach an external model, plus an internal data-controlled model for the same tasks. Then the leak is technically impossible, not merely against policy — and policy alone never stops a well-meaning engineer in a hurry. Full post-mortem →
The pattern: five failures, five missing controls
Line them up and the lesson is unmistakable. A fabricated policy (missing grounding). A $1 car (missing scope boundary). A rogue poem (missing scope boundary plus an untested change). Illegal legal advice (missing grounding and refusal). A code leak (missing egress control). Not one of these was caused by the model being insufficiently capable. Every one was a missing boundary on what the agent could do or say.
That's the useful reframe: an AI agent's safety isn't a property of its model — it's a property of the authority you grant it and the controls you put around that authority. The famous failures are famous precisely because someone shipped an agent with capability and no governance, and the gap found daylight.
What "declaring the boundary" actually looks like
The through-line across all five is that each agent lacked a written, enforceable declaration of what it was allowed to do. That declaration is what the AgentAz specification makes concrete: a small file that states the agent's authority — which tools run automatically, which require human approval, what its worst-case action is, and the resulting trust level. Had the Chevy bot's spec declared "makes no pricing commitments," or the Samsung workflow declared "no external model may receive source code," the missing control would have been a written rule a CI check could enforce — not an assumption that failed in public.
None of this requires a better model. It requires deciding, before deployment, what the agent may do without a human — and enforcing it. You can check where your own agent stands with the free scanner, diff a change for silent authority creep with the drift auditor, or read the full engineering post-mortem library for how each of these failures unfolded and what would have stopped it.
Frequently asked questions
Well-documented ones include Air Canada's support chatbot inventing a bereavement-refund policy that a tribunal held the airline liable for; a Chevrolet dealership's chatbot being prompt-injected into 'agreeing' to sell a $76,000 SUV for $1; DPD's support bot being talked into swearing and writing a poem calling the company useless; New York City's official MyCity bot telling businesses they could break labor and housing law; and Samsung engineers leaking proprietary source code into ChatGPT three times in 20 days. Each traces to a specific missing governance control, not to the model being 'not smart enough.'
Rarely the model's capability. The recurring causes are governance gaps: ungoverned authority (the agent can take consequential actions with no human gate), ungrounded output (it fabricates confident false information), scope escape (it does things far outside its intended function), instruction injection (user input overrides its rules), and unbounded data egress. In every famous case, the failure was a missing control at a specific layer, not a limitation of the underlying LLM.
Define the agent's authority before building it: name its worst-case action, gate the tools that could cause irreversible harm behind human approval, ground factual claims in verified sources rather than generation, constrain scope to intended intents, and enforce those bounds automatically so a later change can't silently remove them. Most famous failures would have been prevented by one specific, nameable control.