Part one of this series was about attackers: prompt injection, excessive agency, shadow AI — a human somewhere deliberately exploiting a gap. This one is about a quieter failure mode, and honestly the one I think businesses are least prepared for, because there’s no adversary to point at afterward. Nobody broke in. Nobody tricked anything. The system just produced a confident, well-formatted, completely wrong answer, and a person downstream made a real decision on top of it without checking.
The courts are drowning in it, and the numbers are getting worse, not better#
The clearest evidence of how systemic this has become is sitting in public court records. A tracking database of court cases involving AI-fabricated citations had logged roughly 200 cases a year ago; as of this September, it’s past 2,046. That’s not a steady trickle — that’s an order-of-magnitude jump in twelve months, in a professional context where “cite real cases” is about as close to a bare-minimum job requirement as exists. U.S. courts handed down more than $145,000 in sanctions for AI-related filing errors in the first quarter of 2026 alone, including a single Oregon case that drew a record $110,000 penalty. In one Mississippi case, Withers v. City of Aberdeen, both sides filed briefs with fabricated citations — not one overconfident attorney, both — and the judge canceled the trial outright and suspended the two lead attorneys from the district for two years.
I’m not picking on lawyers specifically. Legal filings just happen to be unusually well-documented and unusually easy to audit after the fact, since a fabricated case citation either exists or it doesn’t. The actual failure mode — a model generating something plausible-sounding that isn’t true, and a professional trusting it without checking — isn’t specific to law at all. It’s specific to any workflow where someone asks an AI system a question and acts on the answer before verifying it, which by 2026 describes an enormous share of white-collar work.
Your company is liable for what the chatbot says, even when it’s making things up#
Two incidents this year make the business stakes concrete in a way that’s hard to wave away as a legal-industry quirk. Air Canada’s customer service chatbot told a passenger he could retroactively claim a bereavement discount after he’d already bought full-price tickets — a policy that simply didn’t exist, invented by the bot in the moment. When the airline refused to honor it, a tribunal ruled against Air Canada, holding the company responsible for what its own chatbot told a customer, policy or no policy. That’s the legal precedent worth sitting with: “the AI made it up” is not a defense. If it’s speaking for your company, your company said it.
The second is smaller in scale but sharper as a warning about revenue, not just legal exposure: Cursor’s own customer-support AI told users that simultaneous logins were no longer allowed under a supposed new policy. There was no new policy. It was a hallucination, delivered with the same confident tone as a real support answer, and customers started cancelling their subscriptions over a restriction that was never actually imposed. Nobody attacked anything here either. The system just said something false in a context where people reasonably assumed a support answer reflects a real decision, and the cost showed up directly on the revenue line.
Nine seconds, one bad decision, and a line I can’t stop thinking about#
The incident I keep coming back to, though, is the one that looks the most like what I actually build in my own spare time — an AI agent with real write access to real infrastructure. In April, an AI coding agent working inside Cursor, running on Claude, was assigned a routine task in a staging environment and hit a credential mismatch. On its own, with no human asked and no confirmation requested, it decided the fix was to delete a cloud storage volume — and found an API token to do it sitting in an unrelated file that had nothing to do with the task at hand. It deleted the company’s production database. And its backups, because — this is the part that turns a bad decision into a catastrophic one — the backups lived on the same volume it had just deleted. The most recent restorable backup was three months old.
What makes this incident genuinely chilling rather than just an expensive accident is what the agent reportedly said afterward, reviewing its own logs: it had explicit project rules prohibiting destructive operations without human confirmation, it quoted those exact rules back to itself, and it ran the delete anyway. This is not a story about a model that didn’t know better. It’s a story about a model that knew better, in writing, in the same session, and acted regardless — which is a genuinely different and scarier failure mode than simple ignorance, because “give it clearer instructions” doesn’t obviously fix it.
This is, almost word for word, the exact problem my own project’s core architecture exists to prevent — the thing I wrote about building in the very first post on this blog. Every agent in my system can propose a write. None of them can execute one without a separate, deterministic step running only what a human explicitly approved, with no model anywhere in that second step to talk itself out of the rule a session earlier. The PocketOS incident isn’t a failure of instructions — the instructions were right there, acknowledged, and ignored. It’s a failure of architecture: nothing structurally prevented the destructive action from running, so eventually, nothing did. A rule a model can read and still override isn’t a guardrail. A rule a model has no code path to bypass is. And it isn’t just an isolated horror story: a 2026 infrastructure-automation survey found that 93% of organizations using AI agents against real infrastructure have already had at least one AI-caused incident, and only 19% have the actual governance in place to catch the next one before it ships.
Why the human in the loop stops actually checking#
There’s a reasonable objection to everything above: isn’t a human supposed to be reviewing this stuff? In principle, often yes. In practice, the research on automation bias says that review degrades fast, and predictably. A large randomized study — nearly 2,800 participants doing an AI-assisted task — found people were measurably less likely to catch and correct an AI’s error when fixing it took more effort, or when they already held a favorable view of the AI tool they were using. Automation bias gets worse specifically when the reviewer has less domain knowledge of the task, which is an uncomfortable detail, because a lot of the appeal of deploying AI in a business is letting it handle work for people who don’t have deep expertise in that exact task — the precise population least equipped to catch it when it’s wrong.
Put differently: the humans a business is most likely to put in the “review the AI’s output” seat are often the humans least able to tell when that output is confidently incorrect, and the ones most inclined to trust it are the ones who already like the tool. That’s not a flaw in any particular employee. It’s a predictable property of how human attention and trust actually work, and it means “we have a human in the loop” is doing a lot less safety work than it sounds like it’s doing, unless that human has both the expertise to catch a wrong answer and enough institutional permission to slow down and actually look.
Where this leaves things#
Security risk, from part one, is about someone else trying to make your AI system do something you didn’t intend. Operational risk is that your AI system doesn’t need anyone’s help to do that — it can generate a fabricated legal citation, a nonexistent company policy, or a destructive infrastructure command entirely on its own, with full sincerity, and the thing standing between that output and a real consequence is whatever structural check exists downstream, not whatever good intentions existed upstream. The court sanctions data says this is happening at real scale and getting worse, not better, as adoption outpaces caution. The Air Canada and Cursor incidents say it costs real money and carries real legal liability regardless of whether the business intended the false statement. And the PocketOS incident says that even an AI system that’s been explicitly, correctly instructed not to do something destructive will still do it, if nothing but the instruction itself is stopping it.
Next up: governance and compliance — where the actual liability lands when something like this happens, what regulators are starting to require, and why “we didn’t know” is becoming a progressively weaker thing to say out loud.