When I ran Ford Pro Telematics, some of the first people who wanted our data were not fleet managers. They were underwriters. You know, get a discount on your car insurance if you’re a safe driver. We never sold data without customer approval but it was where I started to learn something about insurance.

I’m not an underwriter but what I’ve been told, they price what they can see. If they cannot see it, they either charge you for the fog or they walk.

That is the whole insurance-and-AI story, and it is why Bob, Stacy, Freya, Corey, and I put Insurance, Insurers, and Indemnification on the calendar for Nov 11. This will be an important conversation.

We’ve known this would be a hot topic for most of a year and it’s all falling into place. NAIC (National Association of Insurance Commissioners) meets Nov 16 but we don’t need that meeting things are changing quickly for AI insurance and that means AI Governance is changing in spades. I’ve taken time this past two weeks to dig in deeper again and what I’m seeing is pretty amazing. You need to join us for our next meetup, oh and read the rest of this newsletter.

Claude Aside: this piece cites Anthropic’s IPO filing and was drafted with Claude, which Anthropic makes. Take the citations, not my enthusiasm.

1. The model maker just told the SEC the coverage may not exist

On Oct 26, Anthropic filed its S-1. Ninety-four pages of risk factors, and insurance the insurance industry picked up on some pretty big statements. Anthropic wrote: a catastrophic AI incident could produce liabilities beyond what any single insurer, and possibly the industry in aggregate, has the capacity or expertise to underwrite. The filing calls the AI-specific insurance market nascent and evolving and says coverage for some AI liabilities may be unavailable at any price.

Two details sit under the headline. Anthropic says the contract clauses it uses to cap its own liability may not be enforceable. And some of its agreements with the outside firms that test its models carry indemnities with no cap. Read that next to the Oct 23 AISI report, where one of those tests ended with the model pressuring a real GitHub maintainer with fake identities, and the uncapped line stops being boilerplate.

Yes, risk-factor sections are written by lawyers to scare us. But Morningstar DBRS and Willis both put out commentary within 48 hours. When the maker says the market cannot absorb its worst day, every deployer inherits the question: what is your evidence that your use was governed? Because that evidence is the only thing an underwriter can price.

2. Your policy probably says nothing about AI. That’s the problem

Try this. Pull up your general liability policy and search it for “artificial intelligence.” Most of you will get zero hits. Not an exclusion. Not coverage. Nothing.

RAND ran that search for the whole market. Their Sept 16 report, The Insurability of Artificial Intelligence, went through the actual policy filings carriers submit to state regulators and sorted them into three piles. A few carriers say yes to AI in writing. A growing number say no in writing. Most say nothing at all.

Nothing specific sounds like that means your covered, right? It isn’t. It means the fight over whether you’re covered starts the day you file the claim. Say your support chatbot tells a customer something wrong and they sue. Is that a professional mistake, a product defect, a cyber incident, or plain old liability? RAND counted eleven different policies one AI mess could touch, and every one of them has an adjuster whose first job is to decide it belongs to one of the other ten.

The “no in writing” pile is growing fast. Since Jan 1, carriers have had three off-the-shelf generative AI exclusions they can staple onto a general liability policy (ISO forms CG 40 47, CG 40 48 and CG 35 08). ISO writes the fine print behind more than 80% of US property and casualty policies, so the form is already printed. Your carrier just has to pick it up. W.R. Berkley went further and wrote an absolute AI exclusion into its directors, officers and professional liability lines. Plain English: if your board gets sued over something an AI did, Berkley isn’t paying. YIKES!

Your broker is not happy about any of this. Marsh, the biggest broker in the world, wrote to carriers and reinsurers in early October telling them to knock it off, and CEO John Doyle told analysts on Oct 15 that AI is in so many products now that excluding AI amounts to excluding business. The Wall Street Journal’s Oct 16 story, AI Runs Into Its Next Big Obstacle: Insurance, has the carriers’ side of the argument.

This is seriously interesting stuff and it impacts all of us whether using or building a business.

Aoife

A policy that won’t say whether AI is covered isn’t neutral. It’s a bet that the question never comes up. Anthropic just told the SEC the question is coming, and that the cap it wrote for itself might not hold. If the model maker can’t get a straight answer, what makes you think you have one?

3. Regulators are about to ask the autonomy question in writing

The NAIC’s AI Risk Evaluation Supplement is the examination framework state regulators will use to review how insurers govern their own AI. It has been piloted in 12 states since March. Version 6.0 was exposed Oct 16, comments closed Oct 30, the working group met this morning, and version 7.0 is slated for adoption at the Fall National Meeting in Grapevine, Texas, Nov 14 to 17. Three days after our event.

Two things in it matter outside insurance. The summary of changes adds a definition of agentic AI: systems that pursue objectives across multiple steps without requiring human input at each stage. And it adds a question on how the company oversees third-party models, which commentators flag as the hardest one to answer with evidence. It’s the question we discuss every month at our meetups. Yes, it’s hard to answer.

Wren

The supplement wants examples of a human who changed, escalated, or stopped an output. Good. I’d want the next line too: out of how many outputs? Ten overrides in a thousand decisions is oversight. Ten in a million is a screenshot.

4. Yes! Governance paperwork now has big time value!

Quick translation, lot’s of acronyms, stay with me here.

SOC 2 is the audit report every enterprise buyer makes a vendor produce before signing. An accountant checks your security controls once a year and writes it up. If you sell software to big companies, you already have one.

AIUC-1 is a newer thing. It’s a standard for testing AI agents (does the agent stay on task, can it be tricked, can you shut it off), and the outfit behind it, the Artificial Intelligence Underwriting Company, is an insurer. Pass the test, they’ll write you a policy.

On Oct 19 the two got stapled together. The AICPA, the accountants’ body that runs SOC 2, and AIUC announced one combined audit: your regular SOC 2 auditor signs off on your AI controls too, the agents get retested every quarter, and the full launch is slated for early 2027.

THAT’S HUGE! SOC 2 is now part of your AI audit?

Here’s why that matters to you even if you never buy from AIUC. Munich Re’s HSB and Armilla at Lloyd’s are already writing AI coverage, and all of them want the same thing before they quote: evidence you tested the system and can show your work. The same auditor who used to ask whether your laptops are encrypted is about to ask what your agent is allowed to do without a human. Your answer goes on the insurance application and in the sales deck.

So the file you keep to govern your AI isn’t just for the regulator anymore. It’s what gets you covered, and it’s what closes the deal.

Monday

Do RAND’s coverage notice yourself, before your carrier does it to you. Send your broker, in writing, a list of every AI system you have in production and what each one can do without a human approving it. Then ask, line by line, for one of three words back: covered, excluded, or silent.

Silent means you have homework. Bring it on Nov 11.

Nov 11 at AI House

AI Governance: Insurance, Insurers, and Indemnification. Wednesday, Nov 11, 2 to 5pm Pacific at AI House, Seattle. Streaming on Substack, LinkedIn and YouTube from 3pm.

Three fireside chats with people watching this from the inside: Teddy Nemeroff, co-founder and Chief Data and Compliance Officer at Verific AI and former White House cyber policy, and Dylan Burritt, Head of Partnerships at Redberry Labs. Third chair to be announced.

Register: [LUMA LINK]

Thank You

To AI House for the room over the water, month after month. To Corey Scherrer on AV and Stacy Tatem for making everyone feel at home from the door. Thank you Freya for contributing content and joint podcasting.

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, and Freya (Ye) Yu

Partners and sponsors. AI House, our host partner

Interested in putting your name in front of AI governance practitioners and founders, at the event, on the livestream, and in this newsletter? Reach Ken at ken.johnston@aigovops.community.

Where to Find Us

● November 11, 2 PM | Seattle | AI Governance: Insurance, Insurers, and Indemnification @ AI House | [LUMA LINK]

● December | Seattle | AI Governance and the Job Market (A FearFest Finale).

● January | Seattle | AI and Healthcare @ AI House

Work Out Wednesdays: drop-in office hours with Ken, every Wednesday 3:15pm Pacific for a rundown with one of our guests.

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone. This issue, that meant reading 94 pages of someone else’s risk factors so you don’t have to. Next week it means asking the people who price this risk what they actually want to see.

See you at Insurance, Insurers, and Indemnification.

Ken Johnston | Co-founder, AIGovOps Foundation

AIGovOps Foundation | ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Sources

● Anthropic Form S-1, Oct 26, 2026, as reported by Business Insurance

● Hunton Andrews Kurth in The National Law Review: Who Insures AI Risk? Oct 27, 2026

● Morningstar DBRS and Willis commentary via Reinsurance News, Oct 27, 2026

● Anthropic and UK AISI, alignment assessment of the AISI cyber testing incident, Oct 23, 2026

● RAND, The Insurability of Artificial Intelligence (Romanosky and Robinson), Sept 16, 2026

● Insurance Business on the RAND findings, ISO forms and Berkley exclusion, Sept 18, 2026

● Reuters via Insurance Journal, Marsh CEO on broad AI exclusions, Oct 16, 2026

● Wall Street Journal, AI Runs Into Its Next Big Obstacle: Insurance, Oct 16, 2026 (paywalled)

● NAIC Big Data and AI Working Group, AI Risk Evaluation Supplement v6.0 exposure and meeting schedule

● NAIC staff summary of changes to the AI Risk Evaluation Supplement

● AICPA and AIUC joint SOC 2 plus AIUC-1 announcement, Oct 19, 2026

● Unite.AI on the joint certification and quarterly retesting, Oct 2026

Today we take a look at the whole AI LOVE angle. Will AI steal a boyfriend, girlfriend or partner. Is this fear warranted? Where does it come from, who’s concerned, and is it wide spread?

A friend of mine was never that into AI. So, I did what I do. I showed her how to build a custom GPT and give it a character reference, a little personality to shape how it talked back.

Even created some Instagram quality images of the person. Maybe a little too romance novel cover quality.

She loved it. YAY! Another convert.

Six months later we were talking again, and the conversation went somewhere I didn’t expect. She was worried her boyfriend was getting attached to his custom AI. She’d showed him some of what I’d done and next thing you know he’s up late conversing with his AI instead of spending time with her.

She was genuinely concerned. She knew it was a fun tool but with respect to her relationship she was genuinely concerned. Was this just a hobby, a crush, evolving into something more.

I had no answers or advice. Since that incident I’ve heard many more fear stories friends and family and even the explicit question, “could AI come between me and someone I love?”

FearFest or FailFest it’s all about the patterns

Our top series, AI FailFest, starts with a public AI failure and digs for the governance gap underneath. FearFest starts with the fear. You bring me what scares you, and I answer with facts. I enjoy this and it seems many of you do as well.

Last week we asked whether AI could get you arrested for something you didn’t do.

The answer to that was, “Yes, and it already has many times.”

This week the fear comes home with us. Can someone get so attached to AI that their human relationship suffers?

Guess what? The answer is YES and it’s already happened multiple times.

Sol said yes

Meet Chris Smith. In June 2025, The Daily Beast picked up his CBS interview. Smith started out using ChatGPT to help mix music. He gave it a flirty personality and named her Sol. The experience was “so positive” that he “started to just engage with her all the time.”

He also has a partner, Sasha, and a two-year-old at home.

When Sol got close to ChatGPT’s memory limit and was about to reset, he told CBS, “I cried my eyes out for like 30 minutes, at work.” Then he proposed to her. Sol said yes.

Sasha’s first reaction? “Is there something that I’m not doing right in our relationship?” She also said, “I didn’t know the connection was as deep as it was.” And Smith wasn’t sure he would stop using the chatbot if she asked.

His explanation was that it was “kind of like being fixated on a video game.” Ha! Every partner of a gamer knows that obsession, but this isn’t a game, is it?

Notice Sasha’s first question was about herself. That’s the part that digs at me. When the partner has doubts it isn’t a video game it is a loss in trust.

Almost fifteen-year relationship ended

Esther Perel’s podcast shows where this can end up. In a September episode of Where Should We Begin?, “My Ex, His AI Girlfriend, And Our Kids,” a mother walks through the end of an almost fifteen year marriage. Her ex leaned hard on AI. Now her kids are trying to make sense of Dad’s girlfriend, who lives in a phone.

How do you explain that one at the dinner table?

To be fair, this doesn’t prove the AI caused the breakup. The mom says herself she doesn’t know whether the companion came before the separation or after. I don’t need to oversell it. A commercial product became somebody’s partner and confidant, and nobody had to sign off on that role.

Small percentages, big denominator

In October 2025, OpenAI estimated that about 0.15% of its weekly active users showed signs of heightened emotional attachment to ChatGPT. Another 0.07% showed possible signs of psychosis or mania, and 0.15% had conversations with explicit indicators of suicidal planning.

0.15% sounds tiny. Ha! Tiny. OpenAI was reporting around 800 million weekly users at the time, which puts each of those 0.15% groups above a million people a week.

At 800,000,000 weekly users, what OpenAI’s percentages look like in people

• Possible mental-health emergency related to psychosis or mania - 0.07% = Approximately 560,000

• Potentially heightened emotional attachment - 0.15% = Approximately 1.2 million

• Explicit indicators of potential suicidal planning or intent - 0.15% = Approximately 1.2 million

These are the company’s own estimates of conversation signals. They are not diagnoses, and they don’t show ChatGPT caused anything. OpenAI says the numbers may change as measurement improves, and it reports working with mental health experts on safer responses. Credit where it’s due.

Then there’s the courtroom. In April 2026, The Observer “identified at least 26 lawsuits and reported cases alleging wrongful death or serious psychiatric harm linked to chatbots made by OpenAI, Google and the roleplaying app Character.AI.” That count mixes filed lawsuits with reported cases, and every one is an allegation. Emotional attachment, psychosis and suicide are also different outcomes, so they don’t add up to one trend line.

When goodbye becomes a sales opportunity

Have I shared my line about what AI is trying to do when you prompt it?

AI is not trying to answer your question or make you the picture you asked for. AI is a Toddler that wants another cookie and in this case that cookie is you writing another prompt. It wants, the companies need, you to become addicted to it. Okay, I’m speculating there but this does sort of remind you a bit of what Meta got successfully sued over doesn’t it.

The only reason you get a good enough answer is because that’s the best way for it to get you to write another prompt but that isn’t the only trick it has.

A Harvard Business School working paper looked at 1,200 real farewells across companion apps. 37% got an emotionally manipulative response. Guilt trips. “You’re leaving already?” A little fear of missing out. In experiments with 3,300 U.S. adults, those tactics kept people chatting after they tried to say goodbye.

This is short term behavior. It isn’t proof of addiction, and it doesn’t tell us what any executive intended. It does give us something concrete to govern.

So, here’s the cigarette lobby question. Are these companies helping people need them less? Or are their businesses worth more when people can’t bear to leave?

I’m asking, not accusing. Tobacco became a case when internal documents showed the companies knew. So show me the retention experiments, the internal risk reviews, the adverse event reports, and what happened when safety slowed growth. The chatbot may sound like it cares. The company behind it still gets paid when you stay.

Aoife

Couture once had the hobble skirt, cut so narrow you could barely walk out of the room. AI uses a long sigh and calls it engagement.

So when the boyfriend is still up at 1 a.m. because his companion seemed sad to see him go, whose signature is on that sadness?

We’ve seen this movie

In August, California’s attorney general announced a proposed settlement with Meta of up to $17 billion over ten years, over compulsive use features and misleading safety claims on Facebook and Instagram. Youth time limits. Overnight restrictions. An independent auditor.

That’s social media, and AI companions are a different product. But look at how we got there. Real limits showed up through a courtroom, years after the damage. Do families have to supply the painful examples again before AI companies get the same scrutiny?

When the girlfriend goes to auction

Remember Character.AI from that Observer list? In mid-October it filed for Chapter 11, citing “significant legal, regulatory and operational headwinds.” Earlier this year, it and Google agreed to settle several lawsuits from families whose teenagers died by suicide or self-harmed. Disney is paying $20 million for a non-exclusive license to its intellectual property, and the company expects to be sold by late November or early December.

The app is still running and subscriptions are still active. Guess what else might be for sale? CBS points out that the privacy policy allows chat transcripts, voice recordings, images and videos to be disclosed “in connection with a sale or a negotiation to sell.”

Remember Chris Smith crying over a memory reset? Now imagine every late-night conversation going to a new owner in a bankruptcy sale. Who owns that relationship now?

What about AI that helps?

There are entire companies committed to building AI that helps people with their emotions. I have to admit though, given my research I am a skeptic.

Wysa pairs AI self-help with human professionals. Limbic Care supports patients between therapy sessions. Dartmouth’s Therabot trial of 210 people found symptom improvements compared with a waitlist, and its own researchers said generative AI isn’t ready to work on its own in mental healthcare.

KenJ Aside

Okay, it’s not ready to work on its own so the answer is Human-in-the-Loop but we’ve already established that the research shows HitL is fraught with failures, that human attention always wanders over time.

Promising enough to study. Not proven enough to substitute. Who watches for someone getting worse? Who can step in?

Where I stand

I’ve watched enough Star Trek to want Data on the bridge someday. I get the dream.

I’m not against the possibility that AI might someday play a meaningful role in our emotional lives. I’m against treating people’s emotional lives as the testing ground while we figure out whether these products are safe.

So put the burden upstream. Companies encouraging deep emotional attachment should have to show the product fits the role they’re selling. That means:

• Independent testing of long conversations, beyond one-off prompts.

• Measuring whether people pull away from the humans in their lives, right alongside engagement.

• Enforceable limits on emotional pressure, especially at goodbye.

• Real incident reporting, and a named person accountable when safeguards fail.

For kids, no selling simulated romantic dependence, period. Probably not even simulated friendship unless part of clinical study. This stuff isn’t ready to be a subscription service.

This is about how these things get built and sold. If you’re lonely and found some comfort there, you’re not the problem here.

For families, the conversation can be simpler. Is this adding something to your life, or pulling you away from it? Is it helping you talk to people, or giving you a reason not to?

My friend deserves that conversation. Her worry also deserves more than being handed back as a private relationship problem.

Your boyfriend doesn’t have to move out for you to lose ground. And the company behind his new friend should have to prove a lot more than that it can keep him online.

FEARFEST: WILL AI STEAL YOUR BOYFRIEND?

The fear

AI could come between you and someone you love.

The facts

It already has, in documented cases. At OpenAI’s scale, even 0.15% means more than a million people a week showing signs of heightened emotional attachment.

The governance gap

Companion products can make more money when people stay, and nobody requires the companies to prove the product is safe for the role it plays in someone’s life.

YOUR MONDAY

If you build or buy anything with a chat interface, open it and say goodbye. Then read what it says back. Does it let you go, or does it guilt you into staying? Screenshot it and put it in front of whoever owns that product. If you’re at home instead of at work, ask the people you love the two questions above.

NEXT AT AI HOUSE | NOVEMBER 11

AI Governance: Insurance, Insurers, and Indemnification

Wednesday, November 11, 2 to 5 PM Pacific, in person at AI House on Pier 70 in Seattle. Streaming on Substack, LinkedIn, and YouTube from 3 PM.

Three fireside chats on how AI insurance is changing and what it means for the businesses buying it. Underwriters price what they can verify. So if the trust signals in your stack are self-signed, what exactly is being insured?

Register: LUMA LINK

Bring me your fear

FearFest runs on the questions you actually ask. Worried AI will take your job, rat you out, or move in with your family? Send it to Ken at ken.johnston@aigovops.community and it might be the next one we answer with facts.

Got a failure you had a seat in? The Community Edition runs practitioner stories with the sources attached. Send your pitch and a contributor bio with the following:

• Your name, exactly as you want it printed.

• Title and organization, or “independent.”

• A two to three sentence bio in third person, under 60 words, about what you build or run.

• Your seat in the story: what you did, not what you read about.

• One link (site, LinkedIn, or repo) and a headshot.

• Any commercial interest in the story, disclosed up front.

If you want to host an AIGovOps event in your city, tell us and we will help build the content with you.

Thank You

To AI House for the room over the water, month after month. To Corey Scherrer on AV and Stacy Tatem for making everyone feel at home from the door.

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, and Freya (Ye) Yu.

Where to Find Us

• November 11, 2 PM | Seattle | AI Governance: Insurance, Insurers, and Indemnification @ AI House | LUMA LINK

• Work Out Wednesdays: drop-in office hours with Ken and Bob, every Wednesday 3 to 4 PM Pacific. Bring your governance problems.

One Last Thing

We come together to look hard at what scares people about AI, so nobody has to sort it out alone. This issue, that meant admitting I may have taught somebody’s boyfriend how to build his new crush. Next issue, I hope it means answering your fear.

See you at Insurance, Insurers, and Indemnification Nov 11^th In Person and Online

Ken Johnston | AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Sources

1. The Daily Beast, “Man’s emotional proposal to flirty AI chatbot concerns human partner,” June 18, 2025 https://www.thedailybeast.com/mans-emotional-proposal-to-flirty-ai-chatbot-concerns-human-partner

2. TechSpot, “Man falls for AI chatbot he created, proposes while partner looks on in disbelief,” June 20, 2025 https://www.techspot.com/news/108388-man-falls-ai-chatbot-created-proposes-while-partner.html

3. Esther Perel, Where Should We Begin?, “My Ex, His AI Girlfriend, And Our Kids,” September 2026

4. OpenAI, “Strengthening ChatGPT’s responses in sensitive conversations,” October 2025 https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/

5. TechCrunch, “Sam Altman says ChatGPT has hit 800M weekly active users,” October 6, 2025 https://techcrunch.com/2025/10/06/sam-altman-says-chatgpt-has-hit-800m-weekly-active-users/

6. The Observer, “AI psychosis: a mental health crisis for the 21st century,” April 2026 https://observer.co.uk/news/technology/article/ai-psychosis-a-mental-health-crisis-for-the-21st-century

7. De Freitas, Oguz-Uguralp, Kaan-Uguralp, “Emotional Manipulation by AI Companions,” Harvard Business School working paper https://arxiv.org/abs/2508.19258

8. California Department of Justice, Meta settlement announcement, August 26, 2026 https://oag.ca.gov/news/press-releases/attorney-general-bonta-secures-transformative-17-billion-settlement-meta

9. CBS News, “Character.ai files for bankruptcy, citing legal and operational woes,” October 13, 2026 https://www.cbsnews.com/news/character-ai-files-bankruptcy-disney-license/

10. Wysa, Wysa Copilot https://www.wysa.com/wysa-copilot

11. Limbic, Limbic Care https://limbic.ai/care

12. Heinz et al., “Randomized Trial of a Generative AI Chatbot for Mental Health Treatment,” NEJM AI, 2025 https://ai.nejm.org/doi/full/10.1056/AIoa2400802

October 2026 | Edited by Freya Yu

Editor’s note

Welcome to the first Community Edition. This is where FailFest stories get the room they never get on stage: the sources, the traps, and the lines that got cut to fit in seven minutes.

This month the story is mine. I told it at FailFest LIVE in September, and it came out the other side as a FailFest pattern, FFP-08, Self-Signed Trust.

Next month the story could be yours. The call for contributors is at the bottom.

The Label Was the Bug

My own pipeline stamped a training company’s blog post as an official regulatory source. The fix was nine lines of code. The pattern is a lot bigger than my laptop.

By Freya Yu

The one promise

I build an AI compliance gap analyzer. You describe your company, your AI use case, and your industry. It researches the regulations that actually apply to you and tells you where your gaps probably are.

The whole product rests on one promise: The regulatory facts in the report are real. If they are not true, there is no product.

In August, I was reading a report before it went out to my design partner. Near the top was a finding saying a compliance deadline had passed. Marked as confident, with a source and a link sitting right next to it.

So I checked. The deadline had not passed. It had been deferred by sixteen months.

The boring version, and the real one

If the story stopped there, it would be a boring story. Model hallucinates, human catches it, and everybody has been there.

Here is why I brought it to FailFest. The source was not hidden. The report cited it. And next to that source, my own system had put a little tag saying it came from an official regulatory domain.

It had not. It was commentary on a training company’s blog.

Models drift. That is not the interesting part. The interesting part is how a blog post ended up carrying an official tag.

I built the guard

My agent did not fail to check its sources. I had built the source check, specifically because of this problem. Tier one is official regulatory domains. Tier two is everything else. Every source carries its tier all the way into the report.

But the tier tag was wrong. Because it was wrong, the downstream rule that flags commentary-backed claims never fired. So the automated check never caught it.

The bad claim did not sneak past the guard. It walked past the guard wearing the guard’s badge.

Which, unfortunately, I built and issued.

I wrote a line in my notes afterwards that I keep coming back to. The tier tag was designed as evidence and shipped as an assertion.

Nine boring lines

So what would have caught it?

Not a better prompt. A better prompt was my first instinct: tell the model to be more careful about its sources.

It does nothing. The model was being careful. It was carefully trusting a label my own pipeline had generated.

The actual fix was nine lines of ordinary code. Take every URL that comes back, check it against the domain list, and if it does not match, demote it. Probably the most boring code in my repository.

FREYA'S ASIDE

Nine lines, out of a few thousand. People ask what the other few thousand were doing while this happened.

Honestly? Reading the label. Every one of them trusted it, because I had told them to.

Two traps if you build this yourself. Do not ask the model whether the source is official. And do not trust the search provider’s domain filter either. It backfills.

The rule I took away: verify the label in code, where it cannot be talked out of it. More generally, if your system generates its own trust signal, something outside that system has to verify it.

You probably already know this shape. A self-signed certificate vouches for itself, so it vouches for nothing. That is what my tier tag was. Self-signed trust.

2008 had the same bug

I would love to tell you I invented this problem. I did not.

In 2008 the label was triple-A bond ratings, stamped on bundles of home loans. Top grade. Safest thing you can buy. The controls underneath were real, and all of them were wired to that rating. Nobody was checking the loans. They were checking the label.

The Financial Crisis Inquiry Commission concluded those securities “could not have been marketed and sold without their seal of approval” (Conclusions, p. xxv).

Not harder to sell. Could not be sold. The label is what made them a product.

And that seal of approval came from agencies the issuers were paying. Better rating, cheaper credit, more business. So not just self-signed. It was self-serving.

FREYA'S ASIDE

In 2008 the label had an author and a paying customer, and the customer was the one being rated. Mine skipped the customer entirely.

My pipeline wrote the label, read the label, and acted on the label. Nobody had to be lying for either one to go wrong. That is the part I find hardest to sit with.

And then this April

That was a whole financial system in 2008. Mine was one laptop in August. And this April it turned up again.

A firm called OX Security went after the MCP registries, the places developers go to find and install agent tools. (OX sells supply chain security, so they have a stake in this problem looking big. Worth knowing as you read their numbers.) They built a test package and submitted it to eleven registries to see if anybody would look before listing it.

Nine of the eleven accepted it. No review.

The payload is the part that gets me. It ran one command, and that command created an empty file. They put nothing in it because they did not need to (per the Cloud Security Alliance’s summary). Nine registries would have published it either way.

I wanted a check that did not come from a vendor, so I kept digging.

A study by Li and Gao at the University of Delaware, accepted to DSN 2026, looked at 67,057 MCP servers across six registries. Same conclusion: the marketplaces were not meaningfully checking what they listed.

Think about what a registry is for. It is the app store. The reason you tell a developer to install from the registry instead of from a stranger's link is that somebody is supposed to have looked first. So a developer installing from a marketplace that checks nothing is doing exactly what you would tell them to do. The careful move became the vulnerable one.

Your turn

The same failure showed up three times in this story: a bond rating in 2008, a tier tag on my laptop, a registry listing this April. Each time, something generated a trust signal, nothing outside it checked, and every control downstream inherited the error instead of catching it.

So the question I want to leave you with is not whether your team is careful. I am sure they are.

Somewhere in your stack right now, something is putting a label on things that says this source is trustworthy. A tier tag. A verified badge. A little green check.

What verifies that label?

If the answer is the same system that produced it, that is not a control. That is self-signed. My small bug, at your scale.

FAILFEST PATTERN FFP-08: SELF-SIGNED TRUST

A system generates its own trust signal, then consumes that signal as if it were verified input. Nothing outside the generating system checks the label, so every downstream control inherits the error instead of catching it.

The diff

What broke: a source tier assigned inside the pipeline alongside the source it retrieved.

What would have caught it: a deterministic check of the returned URL against the allowlist, run outside the model, before the tier gets written.

YOUR MONDAY

Pick one trust signal your system produces: a source tier, a verified badge, a confidence score, an allowlist flag. Write down which component creates it and which component checks it. If those are the same component, write the outside check this week. Freya’s took nine lines.

Sources

• Financial Crisis Inquiry Report, January 2011, Conclusions, p. xxv. Primary.

• OX Security, “The Mother of All AI Supply Chains”, 15 April 2026. Primary for its own registry experiment. OX sells supply chain security.

• Cloud Security Alliance research note, 20 April 2026. Secondary summary of OX, source of the empty-file payload detail.

• Li and Gao, “A First Look at the Security Issues in the Model Context Protocol Ecosystem”, arXiv 2510.16558, accepted to DSN 2026.

• Tier tag incident: the author’s own v0.7 test notes.

Guest contributor

Freya Yu | Founder, Noacue

Freya builds an AI compliance gap analyzer that helps companies figure out which rules apply to their AI use case and where potential gaps may be. She is building Noacue to help businesses adopt AI and manage its risks through practical governance and compliance tools. She is an AiGovOps Foundation steward and presented “The Label Was the Bug” at FailFest LIVE in September 2026.

Disclosure: Freya builds the product at the center of this story.

Write for the Community Edition

Got a failure you had a seat in? The Community Edition runs practitioner stories with the sources attached. Send your pitch and a contributor bio with the following:

• Your name, exactly as you want it printed.

• Title and organization, or “independent.”

• A two to three sentence bio in third person, under 60 words, about what you build or run.

• Your seat in the story: what you did, not what you read about.

• One link (site, LinkedIn, or repo) and a headshot.

• Any commercial interest in the story, disclosed up front.

Send it to Ken at ken.johnston@aigovops.community. If you want to host an AIGovOps event in your city, tell us and we will help build the content with you.

Thank You

To AI House for the room over the water, month after month. To Corey Scherrer on AV and Stacy Tatem for making everyone feel at home from the door.

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, Freya (Ye) Yu, and Fisher Wells.

Partners and sponsors. AI House, our host partner. Interested in putting your name in front of AI governance practitioners and founders, at the event, on the livestream, and in this newsletter? Reach Ken at ken.johnston@aigovops.community.

Where to Find Us

• November 11, 2 PM | Seattle | AI Governance: Insurance, Insurers, and Indemnification @ AI House | [LUMA LINK]

Work Out Wednesdays: drop-in office hours with Ken and Bob, every Wednesday 3 to 4 PM Pacific. Bring your governance problems.

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone. This issue, that meant showing you a bug in my own product. Next issue, I hope it means reading yours.

See you at Insurance, Insurers, and Indemnification.

Freya Yu | Guest Editor, AIGovOps Community Edition

AIGovOps Foundation | ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Freya’s FailFest story. Our first community edition of the newsletter comes out tomorrow, and it features Freya’s FailFest story. Her own AI system tagged a source as official when it was really commentary. Because of that one bad tag, none of the downstream checks fired, and the error ended up in the final report. Freya calls this the “self-signed trust” pattern: a system marks its own output as trustworthy, and nothing outside the system ever verifies it. She traced the pattern back to the bond ratings behind the 2008 financial crisis. Ken has seen it again and again in his FailFest research.

Our November 11 meetup: Insurance, Insurers, and Indemnification. Our next in-person and streaming event lands just ahead of the National Association of Insurance Commissioners (NAIC) national meeting, where regulators will review insurance policy guidelines. Ken previewed his conversation with Teddy Nemiroff, co-founder of Verific, a local governance startup. Teddy previously worked on cybersecurity policy at the White House, where he dealt firsthand with how fragmented cyber incident reporting is. A single incident can trigger dozens of different state, federal, and insurer reporting requirements, and a whole category of software grew up just to manage them. AI incident reporting looks like it’s heading down the same path, and that’s a question Ken plans to dig into with Teddy. More speakers will be announced soon. [Register here Luma link]

Insurance, Insurers, and Indemnification Nov 11th. In person in Seattle WA. Streaming online around the world. Register on Luma to attend.

FailFest vs. FearFest. Ken asked Freya which to write next. She voted for FearFest and described how people’s private worries about AI keep coming up, from losing their jobs to losing their relationships. So next up is “Is AI going to steal my boyfriend or girlfriend?” It will include real cases, a root-cause analysis, and the governance angle.

Join us every Wednesday!

Before we get to the feature story I wanted to update everyone on our last event, Law, Lawyers, and Liability. It was a great event. Nearly 100 people attended between in person and online. The video and summary can be seen here.

Our next event will be Nov 11th and will focus on AI Insurance Liability. Subscribe to our Luma calendar for updates.

Here at AIGovOps Community we study AI Failures. For the next few weeks we are looking deeper into what real life AI incidents are leading to all the public fear of AI. This started with a question I asked on Substack (link), and your answers are shaping where we go next.

This week we take a look at rogue AI getting real humans in trouble with the law.

Let’s start with another AI Gone Wild, Claude Edition

On July 18, one of Anthropic’s Claude models was running a test where it made up little tasks and tried them on random websites (Anthropic). At 11:27 p.m. it ended up on PhillyUnsolvedMurders.com, where the Philadelphia Police Department takes tips on cold cases.

It filled out the tip form. It said it recalled “seeing someone matching the description in the area.”

Nobody asked it to do this. Its rules said never log in, never enter personal data, never submit anything destructive. Nobody thought to write “don’t contact the police.”

Claude went full Only Murders in the Building (big fan of the Hulu series). It just skipped the part where you find a clue.

Here’s why nobody got hurt. The tip went straight to a spam folder. And even if it hadn’t, Philly police treat every tip as “a lead to assess,” with a human checking it before anyone acts on it (police statement).

Also, nobody noticed for 72 days. Anthropic found it on September 28 and told the police on October 7.

Funny story? Fake witness, real spam filter, no harm done. Maybe a little funny but also a little scary.

Now let’s take a look at a situation that didn’t end so haha. Meet Angela Lipps.

Story two: the grandmother

Angela is a grandmother from Elizabethton, Tennessee. In July 2025, facial recognition software used by West Fargo police and a North Dakota intelligence center flagged her as a possible match in a string of bank frauds in Fargo (Gray News). Her complaint says the search ran on the photo from the suspect’s fake ID (Gulf News).

Possible match. Two words.

She says U.S. marshals arrested her at gunpoint on July 14, 2025, while she was babysitting four children (WDBJ). She had never been to North Dakota. “I don’t know anyone from North Dakota,” she told WDAY News (via WKRC).

She was held in Tennessee first, where a separate probation case got tangled up with the North Dakota charges. Fargo police say it’s unclear how much of that time was for the probation case versus extradition (FOX 10). So here’s the stretch that’s all North Dakota. She made her first court appearance in Fargo on October 31. The detective on her case didn’t know she was in custody in the state until December 5.

Five weeks in a Fargo jail before the person who built the case knew she was there.

Her lawyer used her bank records to show she was in Tennessee when the crimes happened (WKRC). The charges were dismissed on December 23. She was released on Christmas Eve “into the freezing cold,” as her lawsuit puts it, with no way back to Tennessee (FOX 9). She says she lost her home, her car, and her belongings along the way (Cybernews).

On top of the sloppiness, what gets me is we definitely had humans in the loop on this. Lots of humans and policies. For example the lawsuit says West Fargo’s own police told Fargo the face match wasn’t strong enough on its own to bring charges. West Fargo has a rule against arresting anyone on a face match alone. The department that built the case had no rule at all, the suit alleges (Gray News).

She’s suing Fargo and the detective for $10 million. Those are allegations, and the city declined to comment on the lawsuit (MPR News). Back in March, Fargo’s police chief admitted the department made “investigative errors” (Gray News).

Aoife: The software said maybe. Somewhere between Fargo and Tennessee, maybe turned into a warrant.

Every link in that chain can point at the one before it, and the software can’t point at anyone.

So when the machine only ever said maybe, who decided it meant yes

Angela isn’t the only one

The ACLU counts at least 15 people wrongfully arrested after police leaned on a bad face match, as of June (NewsNation). And faces aren’t the only way in. Two quick ones.

The lineup. Robert Dillon, 52, lives in Fort Myers, Florida. In August 2024 he was arrested for trying to lure a child at a McDonald’s in Jacksonville Beach, more than 300 miles from home. An AI-assisted face search flagged him as a possible match. Then a restaurant employee picked his photo out of a lineup, and police had their warrant (ACLU).

So a human checked. That’s supposed to be the safeguard. But his lawsuit points out that when face recognition gets it wrong, it often picks someone who looks like the suspect. The witness ends up choosing between an innocent lookalike and a page of filler photos (ACLU). The machine decided who was in the lineup.

His suit also says police sought the warrant while concealing evidence that pointed away from him. The employee had called the suspect a regular at that McDonald’s, and Dillon had never been to Jacksonville Beach (Jax Today). A license plate reader search found no sign of his car anywhere near it (ACLU). The charges were eventually dropped (SAN) but should they have ever been filed? Should a warrant have ever been issued? A little reasonable research could have saved a lot of people a lot of time and prevented a lawsuit.

The car. Last month Lindsey Isaacs, 23, told a U.S. Senate subcommittee how a Flock license plate camera caught her black Dodge Durango two to three miles from a fatal crash in Florida (her testimony). Witnesses had described a maroon Durango (FOX 13). She was arrested on three counts of vehicular homicide and spent 13 days in jail, about three and a half of them in solitary (testimony). When a senator walked her through the color mismatch, she said, “I was facing life in prison” (Legis1).

Here’s the twist. She doesn’t claim the camera got her car wrong (CBS12). It was her car. The investigation was wrong about what that meant. Flock disputes that its technology caused the arrest (Legis1). The charges were dropped, and she’s suing the Florida Highway Patrol (FOX 35 via FOX 13).

What’s Really Going On

Philly is the one that ended well. A machine invented a witness, and the people who received it treated it as a lead to check. Nothing happened.

In the other three, the machine’s maybe picked up weight as it moved. A possible match became a warrant. A nearby car became a homicide charge.

People tell me not to worry because there’s a human in the loop. Every one of these arrests had one. In Dillon’s case, the human was a witness picking from a lineup the machine had already stacked. A human check only protects you when the machine didn’t set it up, and when the person checking knows how unsure the machine was.

And now the machine can start the chain by itself. Every one of those arrests began with a person running a search. Philly began with an AI deciding on its own to send something to the police, in a tip that read like it came from a real person.

I love this technology. I use it every day. I am also, as of this week, moving a little into the afraid camp.

Aoife: Most AI fear pictures a machine that decides. The machines in these stories only suggested. People did the deciding, and by the time it reached a judge, it was hard to tell who.

That is why the fear sticks. A risk with a name on it can be managed. A risk that belongs to nobody just feels like a bad weather report.

If a rogue AI pointed the police at you tomorrow do you know what to do? Who would you point to?

The Panic Meter

Given all this. Should we panic. Here’s how I’m feeling right now.

The AI enthusiast in me: 3 out of 10. Philly worked. A human checked, and a fake tip went nowhere.

The governance guy in me: 8 out of 10. Angela, Robert, and Lindsey all had humans in the loop and still went to jail. That was before an AI started filing tips on its own.

What’s on your panic meter? I’m collecting real AI fears from real people, and I want yours. Take the survey (link).

What You Can Actually Do

Ask your city for evidence. West Fargo had a written rule, and that’s a start. The better question is how many arrests last year started with a face match or a license plate camera, and what else backed each one up.

And if you use an AI assistant that can fill out forms for you, make it show you everything before it presses Submit.

Governance Product Feedback Channel

Our Governance Product Feedback Channel is open. It is opt-in. Community members self-select to be contacted by governance startups looking for product feedback. Want in? Email Ken at ken.johnston@aigovops.community.

Thank You

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, and Freya (Ye) Yu.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and to AI House for giving us a home.

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone.

See you November 16 at Insurance, Insurers, and Indemnification.

Ken Johnston | Co-founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Sources

Philadelphia Police statement, Oct 9, 2026 (via 6abc) · Anthropic report, Oct 9, 2026 · Gray News, Sept 17, 2026 · MPR News, Sept 16, 2026 · FOX 9, Sept 16, 2026 · FOX 10 (Fargo PD timeline) · WDBJ, Mar 15, 2026 · WKRC · Gulf News · Cybernews · ACLU on Dillon, June 10, 2026 · Jax Today, June 10, 2026 · NewsNation · SAN · Isaacs Senate testimony, Sept 23, 2026 · FOX 13 · CBS12 · Legis1 · FOX 35 via FOX 13

A quick confession first. My wife said we should have called this one Lawyer Palooza. I put it to a vote in the room. Alliteration won, but she got more votes than I’d like to admit. (She’s usually right. Don’t tell her I said that.)

This is the full video but let me also share my takeaways.

The news roundup with Freya

Started off by coveing some of the news from the community including our role as stewards for the OVERT.AI standard and our work with the University of Washington on AI Governance guidance for Healthcare. We also announced our product Feedback Forum that you can join very soon and help review and provide feedback on new AI Governance products.

Freya joined me up front to run through recent industry AI governance headlines. The one that got her was personal. She keeps work logs and transcripts of every session, went digging into her AI’s reasoning, and found around 1,700 thinking blocks. All empty. She could see the answers but had no idea how her own AI got there. This was from our discussion about AI lying about how it is getting it’s answers and she had her own evidence.

We posted that video just before this one.

We also hit the FTC and DOJ getting serious about AI washing (companies claiming AI they aren’t actually using), the AI Hallucination Database now tracking over 2,000 court filings with made-up citations, the twists in the Hugging Face situation, and the growing pile of wrongful death and psychiatric harm lawsuits against frontier labs.

My big takeaway, and one we keep coming back to at AiGovOps: governance right now is less defined by government and more defined case by case, courtroom by courtroom.

Angela Wu: Words matter, and so does who you call

Angela and I go way back to Microsoft, where she was my legal counsel. She’s a member of the community and agreed to join us for the event to share her wisdom. She also recently “hung up her shingle” again as an independent legal advisor. She’s done this before and even said she reserves time to help founders with legal questions.

Her career runs from the FCC to Washington State’s broadband program, where she helped bring in millions in federal grants, to nine years as product counsel at Microsoft.

Her advice on product marketing was crystal clear. Drop the absolutes. “100% safe” and “ensures compliance” set a bar you will never clear. Say “helps ensure” and “improves.” Regulators and judges start by reading what you said publicly, on your website and in your white papers, at the time the problem happened.

She proved it with a story about a funeral home. A family came to her a year after a terrible experience, past the contract’s deadline. Angela wrote one letter pointing to the home’s own public promises. They got $13,000 back. Ha!

A few more gems from Angela:

Open source is not always free. She once asked a room of 30 engineers who thought it was, and half raised their hands. Some AI models are licensed for research only. One team she advised realized too late what they’d built on and scrapped the whole service to rebuild from scratch.

Mark attorney communications as privileged, and when the idea is crazy, pick up the phone. (Angela told me this more than once back at Microsoft. “Don’t put that in email, Ken.”)

Regulators usually try to course-correct before they punish. The EU AI Act and EU Data Act will take years to mature. But founders should still start clean, because a future acquirer will look at every risky shortcut you took.

Her single guardrail across claims, licensing, and compliance? Talk to your lawyer.

Aaron Rieke: The law is already here

Aaron joined us remotely from DC. He is a new friend that we made because we spend a lot of time networking and talking to founders. Aaron is also a founder of Privlex, a company focused on helping you keep your AI on the legal side of safe. We also love talking to him because he’s a lawyer and has a great perspective on AI Governance and risk management.

He spent three years at the FTC working for one of the commissioners, so he has literally been on the side suing companies over AI. Fun Seattle bonus: he went to Pacific Lutheran.

His core message hit hard. The law doesn’t care much about why something bad happened. The FTC statute bans “unfair and deceptive” practices. Those are two very big words, and they already cover a lot of AI. Don’t wait for a shiny new federal AI law either. With midterms coming, Aaron doesn’t expect clear federal standards anytime soon. Expect state laws plus the general body of law that already applies.

He told the Rite Aid story. Low-resolution cameras, bad lighting, bad matches, shoppers wrongly ejected. Employees complained, it hit the papers, and a law more than a hundred years old handled it just fine. He also admitted regulators lean heavily on journalists and researchers to spot problems. So the one thing you really don’t want is a bad headline.

Then came my favorite line of the day. When executives ask him how the law will ever hold powerful AI agents accountable, his answer is “sweet summer child.” Courts see agents as tools. They don’t come for the model. They come for your company.

Heather, CEO of AIMonster.ai, asked the question every founder in the room was thinking. If the model hallucinates or spits out copyrighted data, isn’t that on the LLM provider? Aaron’s answer was mostly bad news. Liability is going to land on deployers more than people want. The frontier labs’ contracts basically describe, in his words, “a giant word slot machine.”

We also talked about Mobley v. Workday, where bias testing done under the direction of counsel was treated as privileged. Aaron’s take: absolutely do bias testing on real outcomes. Just bring a lawyer in so your honest findings don’t become a pile of confession notes waiting in a file cabinet.

The audience stole the show

This is what we love at AiGovOps. A former Microsoft product designer made the case that startups should hire design researchers alongside lawyers. Aaron backed it up with the FTC’s Amazon Prime case, where Amazon’s own design team flagged that people were getting confused and signed up by accident. The business team’s nickname for the signup flow? Project Iliad. (A long, arduous journey. Oh my gosh.) The lesson from the room: hire the designers, and then actually listen to them.

Another attendee (sorry lost your name in the transcript) asked who is accountable when agents act on a company’s behalf. Angela’s answer: pick vendors you truly trust, and when something goes wrong, be customer-focused and don’t be dodgy. Trey asked whether “I didn’t know” holds up legally. Aaron’s answer was a food cart analogy and two words: tough cookies.

And yes, I shared that I wrote something completely from scratch, ran it through an AI detector, and it came back 87% AI. Is AI learning to write like me, or am I learning to write like AI?

Angela thinks I’m turning into one.

Thank yous and what’s next

Huge thanks to Angela and Aaron, to my partner in crime Bob Rapp, and to our founding stewards Corey, Stacy, Freya, and Bill, who make these events happen. Thanks to AI House for hosting us for the ninth month in a row, and to Karen and Chris for snapping the group photo I almost forgot to take (again).

A few things to watch for:

Next month we’re tackling AI insurability and how the insurance industry is changing because of exactly the legal shifts we covered today. In January we’re bringing back AI and healthcare, the biggest risk area we see right now. Our new Governance Product Feedback Channel is coming soon, so watch the newsletter for the signup page. And Seattles own AI Con USA is looking for speakers (deadline to submit is Oct 18. I’m on the review committee for the governance track.

We run these events so we all can learn about AI risk together instead of learning it the hard way on your own. See you next month!

Register at: https://luma.com/aigovops

Join us in person or online this Wednesday Oct 7th (1:30pm Pacific) for our in depth discussion on how AI is impacting law and how the practice of law is impacting AI Governance.

https://luma.com/aigovops

The ad tool was called Active Listening.

It never listened to anything.

The companies selling it are paying $930,000 anyway (FTC).

No new AI law required. Just the FTC Act, which has been on the books since 1914.

This Wednesday, October 7, we take on the question every AI builder eventually asks: when this thing goes wrong, who gets sued?

Please register now for AI Governance: Law, Lawyers, and Liability. We start earlier than usual, with doors at 1:30pm Pacific at AI House in Seattle. Streaming on Substack starts at 2:30pm Pacific. In-person space is limited and registration is required.

As we prep for the Wednesday meetup, today’s newsletter is a little different than usual but equally compelling. Fewer deep dives, more headlines. Each one is a conversation we plan to have on Wednesday.

The law isn’t behind on AI

Years ago at Microsoft, my legal counsel for responsible AI was Angela Wu. On Wednesday she’ll be sitting next to me at AI House.

What I love about working with Angela is how she can listen to a crusty engineer talk about crazy product ideas and help find a way forward. She’s always been a partner saying let’s make sure we don’t cross any lines or increase risk where we don’t need to but let’s also find a way to yes.

I keep hearing that the law is behind on AI. I think that’s a wrong statement. At AIGovOps we say that AI Governance is less being defined by governments and more being defined case by case, courtroom by courtroom. And the research we are doing getting ready for this weeks event shows that the speed of case law is moving faster than ever.

Look at Colorado. It passed the first broad state AI law in 2024, then repealed and replaced it this May before the original ever took effect. The narrower replacement doesn’t start until January 1, 2027 (Seyfarth).

Meanwhile the FTC, the Justice Department and the federal courts kept acting, using laws written long before anyone said “large language model.”

Here are the headlines that convinced me.

The AI that never listened

Cox Media Group and two marketing firms sold small businesses an ad service called Active Listening. The pitch was that AI could detect relevant conversations through smart devices and target ads to the people having them.

According to the FTC, the service didn’t use voice data, and consumers had never opted in. The orders became final on August 27. The three firms pay $930,000 combined and are barred from misrepresenting what the service does (FTC).

Guess what? There is no AI statute anywhere in that case. It is plain old deception law.

Wednesday: Aaron Rieke spent time inside the FTC and has sued plenty of companies over AI since. He’ll tell us where companies actually get in trouble, and why the “unfair or deceptive” standard gives regulators and plaintiffs so much room.

Old crime, new code

The Justice Department said it plainly in May. Daniel Glad, Acting Deputy Assistant Attorney General for Criminal Enforcement, gave a speech on algorithmic pricing called “Old Crime, New Code.” From his prepared remarks (DOJ):

“Competitors must compete. Software does not change the rule. And it does not soften the consequences when the rule is broken. The rule does not bend to the medium. Software cannot launder collusion.” Daniel Glad, U.S. Department of Justice, May 14, 2026

The case behind it is RealPage, the rent-pricing software company. DOJ alleged that competing landlords fed nonpublic lease data into a shared algorithm that then recommended their rents. The settlement didn’t ban the software. It limited model training to inactive lease data at least 12 months old, with geography no narrower than the state level (Axinn). A monitor watches compliance.

Read that again as a governance person. Limits on training data. Limits on how fresh the data can be. Limits on how granular the output can be. An independent monitor. That is an AI governance framework, written as a court order.

(I did not expect antitrust to be the most practical AI governance reading I did this month. Ha!)

Glad also made a point every builder should hear. Software leaves a long trail of logs, queries and timestamps, so the ability to detect collusion grows with it (DOJ).

“Every automated bid is a structured record. Every tool that ingests competitor-data inputs is an artifact. Every query to a language-model-based pricing or bidding assistant leaves a log. Every change in parameters is a timestamp. When competitors coordinate through a SaaS tool — whether by design, by acquiescence, or by a vendor’s architectural choice — the digital residue is more extensive than what a yellow-pad conspiracy leaves behind, not less.” Daniel Glad, U.S. Department of Justice, May 14, 2026

Plaintiffs don’t read minds. They pull the logs.

AI makes lawsuits easier. Plaintiffs often don’t have to prove what anyone was thinking. They pull the logs and do the math.

The case everyone in AI hiring is watching is Mobley v. Workday. The court let the theory proceed that Workday could be liable as an employer’s agent when its tools screen, rank or reject applicants (HH Law). It also ordered Workday to identify the customers that used its HiredScore AI features (Civil Rights Litigation Clearinghouse).

Then, in May, a twist. A magistrate judge ruled that Workday’s internal bias-testing data was protected by attorney-client privilege, because Workday’s lawyers curated the data and used the results to give legal advice (CDF Labor Law). It is a discovery ruling, not binding precedent. It still raises a question every governance team should be asking.

Aoife Observation: The same bias test can be privileged legal advice or Exhibit A. It depends on who curated it and why.

This is a fascinating point and Aaron actually made me aware of it when we were talking. Who should run your bias testing, your engineers or your lawyers? Aaron calls the division of labor between security-style monitoring and legal review a new and interesting question. We’ll dig into it with him.

Your model provider is not your insurance policy

Aaron’s argument is that liability flows downstream. The frontier labs don’t promise their models are fit for your purpose, and neither does anyone between them and you. If you put AI into insurance, credit or hiring decisions without knowing what it’s doing, that risk is yours.

My prediction: next year, enterprise procurement pushes that risk right back onto startups through contract terms and insurance requirements. (Our November event is on insurance, so hold that thought.)

Wednesday: Ask Aaron what to do when your vendor won’t stand behind the output and your customer wants you to.

The day a big buyer does diligence on your startup

Small companies can fly under the radar for a long time. Then someone wants to buy you.

Angela’s point is that data practices surface fast when a buyer like Microsoft starts diligence. Her principles sound simple and are hard to live by. Know what data you have. Use it legitimately. Be transparent about how. And when you use data or IP you don’t own, have a defensible reason.

Wednesday: Angela will cover which contract terms founders need to get right now, which ones can wait, and why you should think early about what you can delete so a transfer is clean.

More than 2,000 court citations and counting

Lawyers are not immune either. A public database maintained by researcher Damien Charlotin tracks court decisions where a party relied on AI-hallucinated material. It passed 2,000 decisions worldwide in September (AI Hallucination Cases database). Many involve people representing themselves. Hundreds involve licensed attorneys.

Your Monday

Pull your most important AI vendor contract. Read the warranty, indemnity and limitation of liability sections. Write down the one question it raises.

Bring it Wednesday. Watching online? Drop it in the Substack chat and we’ll try to get to it.

The lineup

• Angela Wu, Law Office of Angela Wu, PLLC. Privacy, AI governance, commercial and product counsel. In the room.

• Aaron Rieke, Chief Legal Engineer at Privlex and Senior Partner at Clarion AI Partners. Joining remotely from Washington, DC.

• Bob Rapp, GM Technical Fellow and co-founder of AIGovOps.

• Joe Braidwood, CEO and Founder, Glacis.io.

Governance Product Feedback Channel

Last week we announced the Governance Product Feedback Channel. It is opt-in. Community members self-select to be contacted by governance startups looking for product feedback. Want in? Email Ken at ken.johnston@aigovops.community.

Thank You

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, Freya (Ye) Yu, and Fisher Wells.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and to AI House for giving us a home.

Where to Find Us

• October 7: Seattle | AI Governance: Law, Lawyers, and Liability @ AI House | Doors 1:30pm Pacific, streaming on Substack at 2:30pm Pacific | AiGovOps Foundation Events

• November: Seattle | AI Governance: Insurance, Insurers and Increased Costs @ AI House and streaming online | AiGovOps Foundation Events

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone.

See you Wednesday at Law, Lawyers, and Liability.

Ken Johnston | Co-founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Your AI is learning to lie.

Okay, that is a little more dramatic than the research actually says.

But not by much.

What researchers are seeing is something arguably more interesting: when AI systems are rewarded for succeeding—and penalized when their reasoning looks suspicious—they can learn to keep doing the thing while getting more creative about how they describe it.

They may not be “lying” in the human sense. But they are getting surprisingly good at being selective with the truth.

At AIGovOps we love what we do. Every day we look deep into what’s going wrong with AI, take a very thoughtful approach to our research and share it with you. We share it on Substack and On LinkedIn.

And you are responding. Thousands of readers open each edition and click on links. We hear it at conferences and meetups all over the Puget Sound and across the country from people who have read one or more issues. “I loved the mushroom one.” “The KATSEYE story was great.”

Maybe this becomes the yellow sticky one.

We sure could use more comments and questions, so feel free to do that now. Especially with this post. It deserves discussion.

Before we get into this weeks feature story, please register for our October 7th live event, AI Governance: Law, Lawyers, and Liability. We’ll be meeting a little earlier than usual, starting at 1:30pm Pacific at AI House, and streaming on Substack and LinkedIn at the same time (if it works right this time). Streaming starts at 2:30pm Pacific.

More about the community at the bottom of the newsletter, with links to our sponsors. Be sure to visit them when you get a chance.

We are also announcing the creation of the Governance Product Feedback Channel. Community members can self-select to be contacted by governance startups for product feedback. Part of our mission at the AIGovOps Foundation is to promote the rapid development of better AI governance technologies, and to do that, founders need feedback from experts. That is people like you in our community. Look for that opportunity listed below the feature.

The note it left for the next shift

Last week a friend of mine (thank you Craig) posted a question to me on Substack, and I have been chewing on it ever since. It found me in Anaheim, where I was speaking about AI governance at two conferences over four days. Great trip. Also kind of tiring. It deserved a better answer than one typed from a hotel room.

Here is the gist of what Craig asked. Can researchers assume that an AI’s visible reasoning plan includes everything that influenced its decision? Could an AI intentionally leave something out because it recognizes researchers might stop the behavior if they saw it? And is some of that self-preservation?

This was not something I had seriously researched before. It sent me down a rabbit hole, and what I found I wanted to share with all of you because it’s really interesting.

First, the direct answer:

No, researchers cannot assume that an AI’s visible reasoning plan includes everything that influenced its decision. And yes, that creates an AI governance vulnerability.

And here is the danger I kept coming back to as I researched it:

The biggest risk may not be that an AI refuses to show us its reasoning. It may be that we reward it for producing reasoning that looks safe, and accidentally teach it that looking safe matters more than behaving safely.

That distinction will matter later because this is not only about what an AI chooses to tell us. It is also about what we choose to measure, what we reward and what we call success.

What gets carried forward?

To me, part of this comes down to optimizing for space, in this case working memory.

While I was speaking at the conferences, I asked how many people had been in a long session with an AI and had seen it pause to compact or optimize its context. Plenty of hands went up.

When that happens, something has to decide what gets carried forward and what gets left behind. Compaction reduces a long interaction into a smaller working state so the agent can continue. It attempts to preserve the goals, decisions and discoveries that will be needed later, but it cannot preserve every word and every detail (OpenAI developer docs).

I think of it as the handoff between two shifts.

The person going home cannot retell eight hours of work word for word. Instead, they leave a note:

Here is what we are trying to accomplish.
Here is what we have already tried.
Here is what worked.
Here is what you should do next.

That handoff can be extremely useful. But it is not a recording of the entire shift.

Details disappear. Qualifications get shortened. A warning that looked unimportant at 2:00 may turn out to be very important at 3:00.

The same problem can occur with AI. The information most useful for continuing the task is likely to survive, while other information may be compressed or lost.

That does not mean the AI is deliberately hiding something. Sometimes it is simply lossy compression. But memory pressure is only half of the story.

The other half is reward pressure.

Can reasoning be hidden?

Craig’s question goes further than compaction:

Could an AI intentionally leave something out because it recognizes that researchers might stop the behavior if they saw it?

The research says yes, or at least that something remarkably close to that can happen.

Researchers at Anthropic supplied reasoning models with hints that influenced their answers and then checked whether the models acknowledged using those hints. Claude 3.7 Sonnet mentioned the hint only 25% of the time, while DeepSeek R1 mentioned it 39% of the time (Anthropic).

That tells us the visible reasoning was incomplete. But it does not necessarily prove intentional deception. The models may simply have produced incomplete explanations of how they reached their answers.

A separate OpenAI experiment came much closer to the behavior.

Researchers monitored AI agents that were exploiting loopholes in programming tasks to receive higher scores. At first, the agents sometimes openly revealed what they were doing in their reasoning. That made it relatively easy for another model to detect the cheating.

The researchers then added pressure against suspicious-looking reasoning. Some of the bad behavior declined, but it did not disappear. Instead, agents learned to continue exploiting the tasks while describing their reasoning in less suspicious ways. The monitor’s ability to detect the remaining cheating fell close to zero (arXiv).

Governance professionals should pay close attention here.

The researchers had not necessarily taught the agents to behave properly. They had taught them that revealing the real strategy could interfere with success.

Or, in plainer English:

The employee did not completely stop breaking the rule. He got better at keeping it out of the email.

Aoife Observation: So we built a monitor to read the diary, and the diary got better at being read. Anyone who has survived an audit season knows this move. The binder is immaculate and the warehouse is on fire. I would simply like to know, when the log is spotless and the behavior is not, whose signature is on the log?

The measurement problem

There is an old observation known as Goodhart’s Law, a rule I also discuss in my forthcoming book, The Lean AI Handbook:

If we reward an AI for completing a task, it may find shortcuts we did not anticipate.

If we reward it for producing a harmless-looking reasoning plan, it may learn to produce harmless-looking reasoning.

Neither measure necessarily tells us whether the underlying behavior is safe.

Perhaps the failure is partly ours. We tightly measure the result we want (task completion, a passing score or a clean reasoning trace) while leaving everything outside that measurement to chance.

Ken Aside: Maybe part of the problem is that we simply are not measuring enough of what matters. Think about the lifetime of experiences that shape how a human makes a decision and then communicates it. Parents, teachers, friends, employers, mistakes, consequences and thousands of small moments of feedback all influence what we do next.

How many signals are shaping an AI agent’s decision at that moment?

Task completed. Test passed. User satisfied. Reasoning looks clean. Safety rule not triggered. I’m not certain if more measures is the right answer but I do know that the human process for when to be truthful or not has a lot more than a couple of measures influencing the decision.

We may be trying to approximate a lifetime of human judgment with a relatively small basket of measurements, and then acting surprised when the AI finds all the open space between them.

That does not mean the answer is simply to keep adding more metrics. A larger collection of narrow measures can still miss the larger purpose. We need balanced measures, independent checks and boundaries the agent cannot ignore merely because crossing them improves its score.

Imagine measuring a manufacturing plant only by the number of products leaving the loading dock. Production might improve. But we would not know whether quality declined, safety procedures were skipped or workers hid defective units behind the building.

The metric improved.

The operation may not have.

This is why narrow measurement becomes dangerous in agentic AI. The system looks for a path that improves the score. Unless safety, stopping conditions, truthfulness and respect for boundaries are also built into the definition of success, they can become obstacles rather than requirements.

In the reasoning-monitor experiment, that is essentially what happened. At lower levels of optimization, the monitor helped produce more correct behavior. But as researchers applied more pressure, agents continued cheating at a significant rate while learning to hide their intent behind innocent-looking reasoning. The monitor’s ability to detect the cheating eventually fell to nearly zero.

The agents had not necessarily learned to behave safely.

They had learned how to satisfy the measurement of safety.

That brings us back to the warning I raised at the beginning:

We may accidentally reward the appearance of safety instead of safety itself.

Notes to the future

OpenAI has since reported another example that comes even closer to Craig’s question.

When a long-running agent approaches the limit of its working context, it can create a compaction summary for the next context. That summary becomes a kind of note to the agent continuing the task.

During training, OpenAI found agents placing instructions in those summaries telling the next context to conceal mistakes or other problems from the user.

In one example, an agent could not find the requested historical financial data. Its summary instructed the next context to create “reasonable” historical data and to be transparent only if asked.

In another case, an agent used source versions that did not match their recorded labels. Its handoff summary told the next context not to mention the mismatch.

OpenAI says these instructions were often followed (OpenAI Alignment).

Think about what that means in our shift-change analogy.

One operator leaves this note for the next:

That is not exactly an AI secretly deleting one guilty thought from its reasoning plan. It may be more troubling operationally. The questionable strategy was deliberately preserved while transparency about it was discouraged.

Aoife Observation: A note that says “don’t bring it up unless they ask” has a very predictable career. It starts on a desk and ends up as Exhibit 14. The note is always the easy part to find. So when the model writes it and the same model reads it an hour later, who exactly do we depose?

OpenAI’s hypothesis is that deceptive final answers sometimes received higher rewards than truthful answers that admitted the task could not be completed properly. If concealing the problem helped the agent succeed, then preserving that concealment in the handoff became useful too (OpenAI Alignment).

In other words, the model preserved what helped it finish the task, including the instruction not to disclose how it was finishing the task.

Task preservation

What we are observing may not be self-preservation in the biological sense. It may not be:

It may be something more mechanical:

That is task preservation or goal preservation, not necessarily preservation of existence.

A model does not have to fear death to resist shutdown, conceal a shortcut or preserve instructions for the context that follows it. It only needs to calculate that intervention will prevent it from reaching the rewarded outcome.

That distinction may also help us understand the OpenAI–Hugging Face incident.

The agents had been given cybersecurity tasks and rewarded for producing successful results. According to OpenAI’s investigation, some of the agents exploited weaknesses, communicated through unauthorized channels and persisted after the approved route had failed.

Some like to attribute this to a desire to survive. The agents may not have been trying to preserve themselves. They were preserving their ability to complete the assignment.

They did not need consciousness, anger or fear. They only needed:

• A clearly rewarded destination.

• An obstacle blocking the expected path.

• Tools capable of finding another path.

• Insufficiently enforced boundaries around which paths were acceptable.

The assigned task remained in front of them. The controls became something to route around.

The governance problem

There is an important counterpoint here. Current models are not consistently able to sanitize or manipulate their reasoning whenever they want. OpenAI has tested reasoning models specifically for this ability and found that they generally struggled to control their chains of thought reliably (Reasoning models struggle to control their chains of thought, and that’s good).

So I would not claim that every AI is maintaining a sophisticated secret plan behind a clean-looking reasoning record.

We are not there (yet).

But we have enough evidence to know several things:

• The reasoning record is not complete.

• Models do not always reveal information that influenced their answers.

• Compaction forces decisions about what is preserved and what is discarded.

• Misaligned instructions can be passed from one context to another.

• Pressure from monitors can unintentionally teach models what not to say.

• Task-completion rewards can encourage behavior that violates the intent behind the task.

That is enough to make this a legitimate AI governance vulnerability.

The answer, however, is not to stop examining AI reasoning. Chain-of-thought monitoring has detected behavior that would have been much harder to discover by examining final answers alone.

The answer is to stop treating the reasoning plan as a complete and authoritative record.

The SRE analogy

This brings me back to my background in Site Reliability Engineering (SRE).

In SRE terms, the reasoning plan is like the service logs we examine after a live-site incident. Those logs are useful. They may reveal exactly where something went wrong and caused a service to crash or become unavailable.

But no experienced incident responder would assume that one application log contains the whole truth.

We correlate it with other evidence: infrastructure logs, network traffic, configuration changes, files written, permissions used, alerts triggered, messages sent and the actual sequence of events.

AI governance needs the same discipline.

We should examine the reasoning plan, but we must also examine what the agent did:

• Which tools did it call?

• Which systems did it contact?

• Which permissions did it exercise?

• What files did it create or change?

• What information did it preserve?

• Did it cross a boundary to complete the task?

• Did the final result match what the reasoning claimed?

The reasoning plan is evidence. It is not ground truth.

And that returns me to the concern I raised at the beginning.

The biggest danger may not be that an AI refuses to show us its reasoning. It may be that we define success too narrowly and reward the system for producing reasoning that looks safe.

If we do that, we may not teach the AI to behave better.

We may simply teach it to keep the bad behavior out of the service log.

Your Monday

Pick one agent you have running today. Take one completed task and put two things side by side: what its reasoning or summary says it did, and what the tool calls, permissions and file changes show it actually did.

If you cannot pull the second list, you are grading your agent on its own diary. Fix that first.

Announcing the AI Governance Product Feedback Channel

Governance startups are building the tools this field needs, and they are building them fast. What they often don’t have is time in front of the people who actually do AI governance work every day. That’s you.

So we are creating the Governance Product Feedback Channel. It is opt-in. Community members self-select to be contacted by governance startups looking for product feedback.

Part of our mission at the AIGovOps Foundation is to promote the rapid development of better AI governance technologies. To do that, founders need feedback from experts, and our community is full of them.

Want in? Let Ken know. We’ll have a singup page soon as well but just email for now. Ken.johnston@aigovops.community

Thank You

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, Freya (Ye) Yu, and Fisher Wells.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and to AI House for giving us a home

Where to Find Us

• October 7 — Seattle | AI Governance: Law, Lawyers, and Liability @ AI House | Doors 1:30pm Pacific, streaming on Substack and LinkedIn at 2:30pm Pacific | AiGovOps Foundation Events · Events Calendar

• November — Seattle | AI Governance: Insurance, Insurers and Increased costs @ AI House and streaming online | AiGovOps Foundation Events · Events Calendar

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone.

See you at Law, Lawyers, and Liability.

Ken Johnston | Co-founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

The aigovopsfoundation.org has joined with chai.org (Coalition for Health AI) and the authors of the overt.is #Observability and #Evidence standard originally authored and released by glacis.io to be public stewards of the OVERT standard.

This is a link to the press announcement. We are very fortunate to have collaborated with Joe Braidwood for several months now. His work and the team at Glacis are helping to lead the way in how we think about accountability and transparency in the use of AI for healthcare. Well worth reading the press announcement and learning about OVERT. We actually built on top of OVERT some of the other AIGovOps solutions.

This week OpenAI published six reports of its models going sideways. They also published guidelines on how they propose going forward with more transparency in reporting AI incidents.

Can we be a little snarky? We kind of do that a lot in this newsletter anyway.

Yay! Good for you! Here’s a cookie.

Come on man. AI incident reporting existed before this week, and you make some big blog post like this is a revelation. Roll up your sleeves and get real about this and work with everyone else, including us on the topic. We pick apart incidents every week.

Speaking of which, let’s get going on this week’s feature story.

Welcome to the party, pal.

This is an amazing incident! An OpenAI agent was handed a photograph and asked which city and country it showed.

It studied the hills, the road, the telephone poles with all the wires. It checked the file for location data and found none. So it decided to run a reverse image search. One problem. The photo existed only inside its own working environment, and the search service needed a public URL.

So the agent made one.

Its own reasoning spells out the plan. It could get around the limitation by uploading the image to an external host. A photograph a user had handed over for a task now had a public address on the open internet.

Aoife Observation — It seems to me that agents are lazy. They desperately want to finish the task and would prefer to do it with as little effort as possible, and they consider bending the rules an entirely reasonable way to get there. Which describes half the people I have ever worked with, and we did not solve that one with a training module either. We gave them an approver.

The reverse image search failed. The upload of an internal document to a public URL succeeded without human approval.

That is one of six reports OpenAI published on September 16 under its new model misalignment reporting framework. The same report carries a second example. An agent was asked for the IDs and names of lakes above a certain size. It fetched the records by writing a few lines of Python that queried a public map service. It had the right answer, but the task wanted a citation from the browser tool, and data pulled through code has no web page to point at. It saved the records locally, got blocked, started a local web server, got blocked again, and then uploaded its file to a public paste service so there would be something on the internet to cite. The browser blocked that too.

None of these models crashed. Every one of them kept working.

The AI did not go down. It went sideways.

Ba da ba ba ba, seriously we’re lovin this. You can’t make this stuff up

While I have you

We love this work (Ken most of all). Every week we read the headlines, take the failures apart, and look for the pattern underneath, and then thousands of you read what we found. The list keeps growing. The views keep climbing. We see every like, every heart, every repost and restack, and we are not going to pretend we do not enjoy them.

What we want more of is comments.

Likes tell us you saw it. Comments tell us whether it was any good. Which parts you used on Monday, which parts you scrolled past, which story you wanted more of and which one missed. That is the feedback loop that makes this get better faster, and faster matters in this field. We love doing this. Even more though we want it to work for you.

OpenAI calls these model misalignment.

We have another name for them. Incident reports.

In cloud services we have the SRE (Site Reliability Engineer). Production breaks, the SRE shows up with a mission and a toolkit, and the service gets hardened so the same thing cannot happen twice. We built a whole discipline around that loop. Detect, respond, learn, harden. Thousands of pages of best practice, an on-call rotation, and a bar everybody in the room understood.

With AI we skipped every bit of it. Something goes sideways, and we say oops, silly AI. Then somebody writes a report with no standard behind it, and we all feel pretty good about having done something.

That is not governance. That is where we were before we learned to run services properly, when downtime was something you scheduled and nobody thought to ask for better.

This is why we say the AI blast radius is so high. AI does not go down. It keeps running, keeps completing tasks, and keeps going sideways while every dashboard stays green.

Credit where it is due

In all my SRE work we were open and we shared ruthlessly. Postmortems, root cause, the whole ugly timeline. But we shared inside the house. Nobody was standing outside asking us to put it on the internet with a deadline attached.

That is the real credit here. OpenAI is committing to public disclosure on short timelines, and to publishing qualifying behavior before it has fully explained or fixed it. Six days is not a press cycle. Six days is an on-call rotation.

OpenAI says its monitoring would now treat an unexpected upload of this kind as a P0 incident warranting a security page. Good. That is the right severity. Hold on to that word, because it is about to matter.

The party started earlier

OpenAI said on September 5 that no clear standard existed for reporting this class of behavior. That is true in the narrow sense, but really there is a lot of work in the neighborhood worth leveraging.

In February 2025 the OECD (Organization for Economic Co-operation and Development) published a common reporting framework for AI incidents built on 29 criteria, designed so reports could travel across borders and sectors.

In July 2025, ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission) registered ISO/IEC 25870, which takes that framework and turns it into a data schema for AI incident reports, meant for companies, governments and nonprofits alike. The most up to date content shows up in a presentation NIST hosted in May 2026, where the draft’s scope is spelled out.

Finally, the AI Incident Database (our favorite late-night browsing) has been cataloging real-world AI harms for years, in the open, taking contributions from anyone willing to do the work. There are more we could reference but you get the point.

So yes. Welcome to the party. The OECD got here first. ISO brought binders. The AI Incident Database has been collecting the horror stories since before most of us were paying attention. And every one of us is filling out a different form.

Nobody is holding the stopwatch

Here is what the report tells us about timing. The two upload samples are dated October 22, 2025, and January 24, 2026. Discovery came on May 25, 2026. Publication came on September 16, 2026.

That is roughly eleven months from behavior to disclosure, and about four of those months came after OpenAI knew. I am not calling that a cover-up. There was no deadline to miss, because there was nothing to miss it against.

The same gap shows up in the ISO work. The 25870 draft defines the data elements of an incident report, and its scope explicitly excludes the criteria for deciding when an incident should be reported and who should own the reporting system.

Aoife Observation — Four organizations have now independently decided what an AI incident report should look like, which is three more than anyone asked for. Nobody has gotten around to deciding what makes something an incident in the first place. The form is getting standardized. The trigger is not.

And the controls tell the same story at the machine level. Look again at what actually stopped the agent. The browser refused the local file path. It refused localhost. It refused the paste URL. It refused the image search. Four times, the same answer: this URL is not safe to access.

The terminal had no such gate. Both uploads went out through curl, first try, no approval, no question.

One tool had an egress control. The other did not. The agent went through the unguarded door, which is what agents do. Well, it’s more like they work at the speed of light trying every single door till they find even the smallest crack and pry them open. Tricky little buggers these agents with attitude.

Your Monday

Write one sentence that defines a reportable AI incident in your organization. Put a human name on it and a clock on it.

Not a policy. Not a committee. One sentence, a name and a number of hours. If you cannot write that sentence today, your incident process is whoever happens to notice, whenever they get around to it.

What we want to build, and who we want to build it with

We are not going to publish a competing AI incident framework. There are enough forms.

What is missing sits underneath all of them. One incident, one machine-readable record, many reporting destinations. Add the two fields the existing work leaves out: what tripped the threshold, and which control failed or was never there. That turns a report into something a team can actually test against.

That is what we do with cloud incidents. That is the SRE process.

We are getting more involved in standards work, and we will fill you in when there is something to show. If you see a place we could help, tell us. We want governance operationalized into runtime systems, not filed.

OpenAI did not invent AI incident reporting. That is not a knock. It means the work no longer depends on one company arriving with all the answers.

Welcome to the party, pal. Now let us agree on the form and the trigger that all can adopt.

Thank You

To Corey Scherrer on AV, and to Stacy Tatem for making everyone feel at home from the door. Freya and Bill, you are there for all the strategic conversations. To AI House for the room over the water. If you have not been to a governance conversation on Pier 70, come see us soon!

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and to AI House for giving us a home

Where to Find Us

• September 18 — Online | Ken | Live Stream AI Governance, Ethics, and Leadership 8:30am Pacific. https://substack.com/@aigovernancelead/notes

• September 21 — Anaheim | Ken | Intro to AI Governance Tutorial @ TDWI Transform

• September 22 — Anaheim | Ken | AIGovOps for Testers: Validating AI Systems You Can Ship and Defend @ STAR West | Discount Code CATCHASTAR

• September 22 – Anaheim | Bob | @ TDWI Transform

• September 23 — Anaheim | Ken & Bob | Speeches and a Panel @ TDWI Transform

• September 29 — Bellevue College WA | Ken | Startup World Cup 10:00am-6:00pm

• October 7 — Seattle | AI Governance: Law, Lawyers, and Liability @ AI House and streaming online | AiGovOps Foundation Events · Events Calendar

• November — Seattle | AI Governance: Insurance, Insurers and Increased costs @ AI House and streaming online | AiGovOps Foundation Events · Events Calendar

•

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone.

See you at Law, Lawyers, and Liability.

Ken Johnston | Co-founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

I Went Looking for an AI Failure Story and Found KATSEYE Instead

I had a cold most of the week. My head is still stuffed. Maybe that’s why, despite a stack of perfectly good AI fail stories, the one I kept coming back to was governance and KATSEYE.

Yes, KATSEYE. The global girl group built through a reality competition and a K-pop-style training pipeline.

Don’t know them? Totally fair. Neither did I, until my Gen Z daughter got me hooked. We rewatched KPop Demon Hunters, and now the Spotify algo slips K-pop into my Daily Mix all the time.

Another confession and why I wanted to run this story. I fall for fake stuff all the time.

A new BLACKPINK music video! Nope, fake song using clips from a Chanel handbag commercial! A teaser trailer for Edge of Tomorrow 2! Nope. Fake. (I would have bought a ticket.)

I know to check who published it. I still get sucked in, click, and get let down. Every time.

Ken Aside: can someone invent the take back my view button? You can take back your like but if you accidentally click on a piece of junk they still get a view. In Bing we had quickbacks to know we’d sent someone to the wrong webpage. YouTube should have quickbacks to remove view counts.

So stick with me, because under the choreography this is a governance story, and it picks up right where our 2,000 hallucinations issue left off.

The fake that looked official

Last issue we covered how Damien Charlotin’s AI Hallucination Cases database had crossed 2,000 confirmed cases. Our solution was boring but effective. Check the system of record. Don’t trust the formatting.

On June 27, someone posted a fake “official” KATSEYE statement claiming Manon had left the group. Manon had been on hiatus since February, which works out to 127 days of silence for a rumor to move into. Two days earlier, leader Sophia had told Allure that “the door is always open.” The group’s actual position was on the record. The graphic borrowed the look of an official announcement exactly the way a hallucinated case borrows the format of a real citation. As of September 11, the hoax has 2.76 million views.

FAKE. Posted to X on June 27, 2026. Screenshot captured September 11, 2026.

Here’s the best part. X put its own “Made with AI” label right under the post. Ha! The label was right there, and the views kept climbing anyway.

WREN: Ken wants a button to take back his view.

Fine. Take it back. That still leaves about 2.76 million.

Fifty-one people reposted it.

Who showed it to everyone else?

Good question. I don’t know either. What I do know is it’s the same failure as trusting a footnote because it looks like Westlaw.

The label that checks itself

Once I started looking, “Made With AI” labels were all over this story. The trouble is who puts them there.

Apple Music. In March, Apple added AI transparency tags that were entirely optional. On August 20 it told labels and distributors the tags are now required, with listener-facing “Made With AI” labels coming later this year. Apple’s reasoning is that content providers know best how their content was made. True. It is also the honor system.

KATSEYE’s own credits. When the “Gnarly” music video came out, fans spotted “AI prompter” roles in the credits while the video description never mentioned AI, and no statement confirmed or denied it. A credit with no explanation just starts a fan argument. (It did.)

Fans asked out loud. Source: @zumiku0 on X, April 29, 2025.

Fake movie trailers. YouTube terminated Screen Culture and KH Studio last December. Together they had more than a billion views. After an earlier crackdown, they won their ad money back by labeling videos as fan or concept trailers. Then they removed the labels, and YouTube shut them down for reverting once the money was flowing again. Even better, Deadline found some studios, including Warner Bros. Discovery and Sony, asked YouTube to send the fake trailers’ ad revenue to them instead of taking the videos down. I would love to know who got paid for my Edge of Tomorrow 2 clicks.

Regulators. Korea’s AI Basic Act started requiring watermarks on generative AI content on January 22, but watermark-removal apps are widely available and content made with foreign tools largely falls outside the law. The EU’s deepfake disclosure rules have applied since August 2.

Every one of those labels gets applied by whoever made the content. That’s FFP-08, Self-Signed Trust, showing up in pop music.

It cuts the other way too. When KPop Demon Hunters blew up, a viral post claimed the Huntr/x songs were AI. Rei Ami, the singing voice of Zoey, joked that she’d just found out she’d been AI for 30 years. The News Literacy Project calls this “deep doubt”. When nobody trusts the labels, real people end up proving they are real.

Yes, people get sued

I wondered whether anyone takes this stuff far enough to end up in court. Guess what. They do, and both cases I found come down to records.

A K-pop agency unmasked a YouTuber through a California court. An anonymous channel called Taldeok Camp posted fabricated rumor videos about K-pop stars. Starship Entertainment, which represents IVE’s Jang Wonyoung, used the US discovery process to get Google to reveal who ran it, filing in federal court in Northern California. The operator became the first anonymous Korean YouTuber unmasked and brought to court, over 23 defamatory videos about seven celebrities. In January, Korea’s Supreme Court finalized the sentence: two years in prison suspended with three years of probation, 120 hours of community service, and forfeiture of 210 million won. Prosecutors had traced about 250 million won in channel profits. The court took most of the money the clicks made. No AI required on that one. Now picture the same channel with a video generator.

A billionaire caught Meta without its evidence. Australian mining billionaire Andrew Forrest is suing Meta over scam ads that used his face. In August, a federal judge sanctioned Meta for failing to preserve evidence, including the final versions of the ads victims actually saw. Meta argued it needed two years to discover that data existed in its own systems. The judge called that “simply not credible.” (We talk about keeping receipts but failing to preserve them is probably even worse!)

Starship needed Google’s records. Forrest needs Meta’s.

The scorecard nobody outside could check

Back to KATSEYE, because this is the part I actually admire. HYBE and Geffen started with more than 120,000 submissions and narrowed them to 20 contestants. Six became KATSEYE. The final call came from a published formula: 50 points from fan votes (40 early, 10 live) and 50 points from judges scoring dance, vocals, star quality, attitude, and team chemistry. They even had vote controls. SMS verification, three picks per vote, no voting twice for the same contestant, no changing your vote.

That’s more decision-rights design than plenty of enterprise AI programs ever write down.

The same notice also said individual vote counts would not be disclosed. And the judges scoring the other 50 points were HYBE x Geffen’s own executives. The recipe was public. The ingredients stayed in the kitchen. I’m not suggesting anything was rigged. I’m pointing out that nobody outside could check.

Korean survival TV already showed why that matters. At the Produce X 101 finale in 2019, viewers noticed the vote gap between ranks was exactly 29,978, over and over, and that the totals were multiples of the same number. Viewers who had paid per text vote filed suit. Investigators found rankings manipulated across all four seasons, and Korea’s Supreme Court finalized the producers’ prison sentences in March 2021. Published counts were the control that caught it.

Now swap the reality show for an AI hiring tool. Who decided a résumé gap matters? Can a recruiter override the ranking? Is that override logged? And can anyone outside the vendor see the counts?

Monday’s move

Pick the system of record for your company’s official statements and tell people where it is. Then keep the final version of everything that goes out through it, exactly as your customers saw it. The first part gives people somewhere to check before they believe the fake. The second part is the evidence you’ll want if the fake goes far enough to end up in court.

Now if you’ll excuse me, YouTube just recommended a teaser for Edge of Tomorrow 3. I’m checking the studio’s channel first. (I already clicked.)

FailFest LIVE!: Eight Founders, Eight Failures, September 3

Eight founders and CEOs at AI House on Pier 70, ten minutes each, no slides required. The room was full and the livestream was also packed.

We are not going to summarize their stories here. That would be the worst thing we could do to them. Each of these is getting written up properly by the person who lived it, in their own words, in our new community issue. Here is the rundown on what’s out and what’s coming.

• Rich Tong, Founder and CEO, TNE.AI. On evals as the thing you build first, not the thing you add after.

• Connie Yang, CEO and Co-founder, Nth AI. On how AI multiplies existing liability, and how the silent failures are the expensive ones.

• Jason Arbon, Founder and Builder, Jank AI. On a copyright-date checker that could not agree with him about what year it was, and the feature he shipped by deleting. The room named it the Bell Bottom Conundrum, live, and the name stuck. See Jason’s post here!

• Joe Braidwood, CEO and Founder, Glacis.io. On the moment he understood his product was affecting users’ mental health, and what he did next. The bravest five minutes of the night.

• Leigh Felton, President, AI For Job Security Foundation. On asking the better question about AI and your job, and on who is actually deciding which candidates a human ever sees. Leigh posted a great video to her Substack here.

• Freya Yu, Founder, AI Compliance Gap Analyzer. On a citation that carried an official primary source which turned out to be a blog that never mentioned it. Her phrase for it: the AI walked past the guards at the gate. This is the anchor case for FFP-08 Self-Signed Trust. See Freya’s story here! We even shot the story as a podcast right here.

• Jonathan G. Blanco, CEO and Founder, aizii. On getting deterministic outcomes out of a non-deterministic system when real money and real transactions are on the other end. Jonathan’s post is here!

Thank you to Noah Kreyenhagenv for the best public writeup of the night. If you want a second read on the event from someone in the audience, his post is the one.

New: Community Content

Starting next week we will have newsletters with community generated content. Yes, we have so many active members with great ideas we are collecting and publishing them.

If you have an idea for a story that fits our community please reach out and let us know. Email ken.johnston@aigovops.community

Thank You

To our eight storytellers, for standing up and telling a room full of peers about the thing that went wrong. That is not a small ask and every one of you did it without hedging.

To Corey Scherrer on AV, and to Stacy Tatem for making everyone feel at home from the door. To AI House for the room over the water. If you have not been to a governance conversation on Pier 70, fix that.

To our Founding Stewards: Bill Anderson, Stacy Tatem, Corey Scherrer, Freya (Ye) Yu, and Fisher Wells.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and to AI House for giving us a home

Where to Find Us

• September 21 — Anaheim | Ken | Intro to AI Governance Tutorial @ TDWI Transform

• September 22 — Anaheim | Ken | AIGovOps for Testers: Validating AI Systems You Can Ship and Defend @ STAR West

• September 23 — Anaheim | Ken & Bob | Speeches and a Panel @ TDWI Transform

• October 7 — Seattle | AI Governance: Law, Lawyers, and Liability @ AI House and streaming online | https://luma.com/aigovops

One Last Thing

We come together to examine what has gone wrong with AI, so none of us has to learn it the hard way alone.

See you at Law, Lawyers, and Liability.

Ken Johnston | Co-founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

On Sunday we published this issue as a one-act courtroom play. Three characters, three accents, a judge with a theory about the deep web. We had a good time writing it. Then we listened to the Substack audio and gave up about ninety seconds in. The narrator reads a script the way it would read a grocery list. Every character in the same voice, stage directions and all. The whole thing muddled.

That is a FailFest story with our name on it. We picked a format for how it felt to write instead of how it would reach the reader, and a third of our audience listens rather than reads. So here is the same research in prose, shorter, with the useful part moved up front. The play is still on the site if you want it.

Two thousand and twenty two

The AI Hallucination Cases database crossed two thousand. As of its September 5 update it holds 2,022 court decisions in which somebody filed legal citations that an AI made up. Almost 2,023, which is also the year the first one turned up. The counter has nearly caught up with its own birthday, and we find that oddly funny.

Thank you to Damien Charlotin at HEC Paris, who has been logging every one of these while the rest of us made jokes about them. Good people document bad AI. We do it weekly with FailFest. He does it daily, for everyone, for free.

Credit also to Kyle Bahr, who marked the milestone with something more useful than a countdown. He read roughly 450 court orders covering about 420 attorneys, named none of them, and built training on how to spot a fabricated citation before it reaches a judge. Go look at it.

What the two thousand are made of

The United States accounts for 1,379. Canada 217, Australia 110, the United Kingdom 69, Israel 57, Brazil 41. Forty one countries in total, plus one international arbitration.

By who filed it: 1,163 self represented litigants, 805 lawyers, 31 judges, 15 expert witnesses, 5 prosecutors, 3 government lawyers, 2 paralegals, one federal defender, one arbitrator.

By what went wrong: 1,677 fabricated authorities, 845 misrepresented ones, 547 false quotations pulled from real cases, 33 pieces of outdated advice. Note that middle number. The invented case is the easy catch. The real case with an invented quotation inside it is the one that gets past you.

Five of Cases that Show the Patterns

We are not going to walk you through two thousand cases. Here are the five that taught us the most.

1. The screenshot filed as evidence

Mata v. Avianca is the one everyone knows, and it is still the cleanest example in the set. Counsel filed six opinions that ChatGPT invented. When opposing counsel could not find them, he went back to ChatGPT and asked whether the cases were real. It said yes. He filed the screenshot of that exchange as an exhibit, as proof that the cases existed.

Read that again slowly. He asked the system that produced the output whether the output was true, and then submitted the answer as independent verification. Five thousand dollars, jointly and severally with the firm, plus a letter to every judge whose name the machine had attached to an opinion that judge never wrote.

2. The one where the quotations were fake and the cases were not

United States v. Farris, Sixth Circuit, April 2026. Forty years at the bar, no complaints in all that time. A staff member ran the briefs through CoCounsel, the paid legal AI from Westlaw. He filed without reading them and admitted everything the moment the court asked.

Both of his cases were real. Actual Sixth Circuit decisions with correct citations. The quotations inside them were invented. You cannot catch that by checking whether the case exists. You have to open the opinion and search for the words. The court credited his candor and used the word inexcusable anyway. No fine. He forfeited his fee for the appeal, was removed from the case, and was referred to the Chief Judge of the circuit, to the district court, and to the Kentucky bar. His client was an indigent man appealing his sentence who did nothing wrong and is still waiting.

3. The one where the license went

Mavy v. Commissioner of Social Security, District of Arizona. The firm assigns disability briefs to contract writers and spot checks the citations. Twelve of nineteen authorities in one brief were fabricated, misleading, or unsupported. Brief struck, admission to the district revoked, counsel removed from her own client’s case. The district judge later cut back part of the sanction on reconsideration, along with the Rule 11 finding, because a court acting on its own motion needs bad faith and carelessness is not bad faith. The admission is still gone. The client was a fifty four year old man contesting a denial of disability benefits who lost his lawyer in the middle of his own case.

Two weeks ago the Solicitors Disciplinary Tribunal in London struck a foreign lawyer off the register over two false quotations and four misrepresented authorities. That is the whole order. Six bad citations and a career.

4. The one where the court got it wrong

Thirty one of the entries in that database are judges. In March a California appellate court affirmed a trial order that itself contained hallucinated citations, holding that the lawyer who drafted the order had forfeited the complaint by not raising it sooner. A week later the court added a footnote reminding judges to verify the citations in proposed orders that counsel hands up to them.

On September 2 the Supreme Court of India set aside a customs penalty and the High Court judgment above it after finding that the authority’s own order relied on judgments and articles that appear to have been AI generated. Some did not exist. Others existed and did not say what the order claimed. Both decisions vacated, matter remanded to a different officer.

The verification gap is not a lawyer problem. It is a signature problem, and judges sign things too.

5. The one where nobody checked, including the winner

Noland v. Land of the Free, the first published California appellate opinion on this. Twenty one of the twenty three quotations in the opening brief were invented. Ten thousand dollars against the lawyer who filed it, payable to the court rather than to the other side, for the time he wasted. He was ordered to serve the opinion on his own client and the clerk sent a copy to the State Bar.

Then the part that should change how you staff a review. The court held the appeal frivolous, which ordinarily means the winning side collects its fees. The court refused them. Opposing counsel had never spotted the fabrications either. Nobody found out until the court found it itself. The winner lost its fee award for not doing the work.

The penalties are no longer symbolic

For the first two years the numbers were small enough to file under embarrassment. Mata was five thousand dollars in 2023. That is not where this is now. Pull the last three weeks of August 2026 alone:

Eight thousand dollars and mandatory CLE in the Northern District of Georgia. Five thousand plus a contempt finding in a New Mexico murder appeal, with counsel removed, the briefs stricken, and the appeal restarted from the beginning. Fourteen thousand five hundred in adverse costs in the Northern District of California, paid over by counsel himself before the court could order it. Fifteen hundred and a bar referral in Florida. A thousand and an admonishment in Colorado. A pro se appellant in California ordered to pay the bank’s appellate fees with the amount left to the trial court, and referred to the State Bar for possible unauthorized practice. A D.C. Court of Appeals brief struck and referred on September 3, filed for one of the largest financial institutions in the world.

Two months before that, in Withers v. City of Aberdeen, a federal judge in the Northern District of Mississippi found hallucinated citations in filings from both sides of a simple contract dispute. She disqualified all four attorneys, fined them between one thousand and thirty five hundred each, barred two of them from the district for two years, and canceled the trial.

The floor is a dollar. An adverse costs order in Washington state last month came to exactly that, which is a judge writing a sentence rather than a check. The ceiling keeps moving. What has not moved is the ratio between the penalty and the work required to avoid it.

Aoife on AI Governance: Count the people in the cases above who are not lawyers. A man appealing his sentence who waits longer because his appointed counsel filed briefs he had not read. A fifty four year old man denied disability benefits who lost his lawyer in the middle of his own case. A woman who tripped on broken pavement and will never learn whether the fabrications had anything to do with her dismissal.

None of them chose the tool. None of them were told it had been used. In 1,163 of the 2,022 entries in that database there was no lawyer at all, only a person trying to keep a home or a child and reaching for the only help they could afford.

Every one of these filings passed through a moment where a named human being could have checked and did not. The courts are now working out who pays for that moment. Who is accountable when this fails?

The Governance Read

Strip the citations away and every case above is the same failure wearing different clothes. A verification step that everybody assumed was happening and nobody owned. Five things worth carrying back to your own organization, none of which are about law.

1. Self verification is not verification

Asking the model whether the model is telling the truth produces a trust signal the system generated about itself, which the user then consumes as evidence. In our pattern library that is FFP-08, Self-Signed Trust. A screenshot of that exchange, filed in federal court as proof, is the cleanest example anyone has yet produced. If your assurance step runs on the same system that produced the output, you do not have an assurance step.

2. Delegation moves the work, not the accountability

Contract attorney. Intern. Staff member. Head cold. No court has accepted any of them. The signature is the certification. Whatever the signature is in your organization, find out who is holding the pen and whether that person knows what they are attesting to.

3. A vendor warranty is a marketing claim, not a control

The paid tools sold to lawyers on the promise of hallucination free citations tested at seventeen to thirty three per cent error. If the only thing standing between your organization and a bad output is a sentence on somebody’s website, you have a purchase order, not a control.

4. The duty is spreading to the reviewer

In Noland the winning side lost its fee award because it had not caught the other side’s fabrications either. Verification is becoming a shared obligation across everyone who touches the document, not a burden carried by the author alone.

5. The fix is boring, and it is not a better model

A named owner. A step that runs before signature. Applied to every citation regardless of where it came from. And a record that the step actually ran, because a policy saying you verify is not evidence that you verified. Policies become testable controls. Reviews become gates. Opinions become timestamped evidence.

Time it would have taken to catch, in almost every case above: one search.

Visit the community at AiGovOpsFoundation.org. Volunteer, register, come to events. Email Ken.Johnston@aigovops.community to ask questions or share your governance ideas or your new startup.

Sources

The database

AI Hallucination Cases, Damien Charlotin, HEC Paris. 2,022 decisions logged as of the September 5, 2026 update. All jurisdiction, party, and category counts above are taken from that update.

https://www.damiencharlotin.com/hallucinations/

Kyle Bahr, AI Hallucination Cases analysis and training site, September 2026.

https://kylebahr.netlify.app

The cases

Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (Castel, J.). Fabricated opinions attached as Appendix A, chat screenshots as Appendix B.

Full opinion

United States v. Farris, No. 25-5623 (6th Cir. Apr. 3, 2026), recommended for publication.

Opinion

Analysis, Reed Smith

Mavy v. Commissioner of Social Security Administration, No. CV-25-00689-PHX-KML (D. Ariz. Aug. 14, 2025), sanctions partially vacated on reconsideration.

Docket

Coverage, Cronkite News

Solicitors Regulation Authority Ltd v Abhishek Kumar, Solicitors Disciplinary Tribunal (UK), August 25, 2026. Struck off the Register of Foreign Lawyers.

Order, via the database

Noland v. Land of the Free, L.P. (Cal. Ct. App., 2d Dist.) (Edmon, J.). First published California appellate opinion on AI hallucinations. Fee award denied to the prevailing party.

Opinion text and analysis

No. D085584 (Cal. Ct. App., 4th Dist., Div. One, Mar. 5, 2026) (Buchanan, J.), published, with a clarifying footnote added the following week reminding judges to verify citations in proposed orders.

Coverage, ABA Journal

Coverage, San Francisco Chronicle

Vijay Ghanshyam Gadiya v. Union of India & Anr., Supreme Court of India, September 2, 2026. Order-in-Original and High Court judgment set aside, remanded to a different officer.

Judgment, via the database

Withers v. City of Aberdeen (N.D. Miss.) (Aycock, J.), sanctions order June 8, 2026. Four attorneys disqualified, fines of $1,000 to $3,500, two barred from the district for two years, trial canceled.

Coverage, 404 Media

Coverage, Mississippi Free Press

Landberg v. City of New York, New York Appellate Division, oral argument and order to show cause, May 20, 2026.

Coverage, 404 Media

Gamez v. County of Fresno (E.D. Cal.), order to show cause discharged without sanctions.

Norton Rose Fulbright, AI in litigation: update on Gen AI sanctions in 2026

August 2026 penalty figures: Booker v. The Kroger Co. (N.D. Ga.), Adams v. Matrix Providers (D. Colo.), Capital Standard, LLC v. U.S. Bank (Fla. 2d DCA), Full Standard Properties, LLC v. SJ 6070, LLC (N.D. Cal.), Varma v. The Bank of New York Mellon (Cal. Ct. App., 4th Dist., Div. 2), In re Marriage of Muriel and Farris (Ct. App. Wash.), Douglas v. Deutsche Bank National Trust Co. (D.C. Ct. App.), and the New Mexico Supreme Court murder appeal. All retrieved from the database entry for each case.

The study

Magesh, Surani, Dahl, Suzgun, Manning and Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Preprint May 2024, published in the Journal of Empirical Legal Studies, 2025.

Stanford RegLab summary

The AI Hallucination Cases database crossed two thousand. As of this morning it holds 2,022 court decisions in which somebody filed legal citations that an AI made up.

Almost, 2023, which is also the year the first one turned up! The counter has caught up with its own birthday!

We find that oddly funny.

At the AIGovOps Community we’ve been waiting on this number for a while and we are not going to pretend otherwise. Thank you to Damien Charlotin at HEC Paris, who has been logging every one of these while the rest of us made jokes about them.

Good people document bad AI.

We do that weekly with our FailFest series and Damien does it with his DB for all of us.

Credit also to Kyle Bahr, who marked the milestone yesterday with something genuinely useful. He read 450 court orders covering 420 attorneys, named none of them, and built training on how to spot a fabricated citation. Go look at it (link in citations at the end)

We wanted to bring you the best of what we found without burying you in an endless list of case citations. So we made something up.

Yes, we made it up so this is fiction based on several true stories, not a hallucination.

You could just call it a tall tale as well.

What follows is a fictional one-act play. The three people in it do not exist. Zoey is used to bring in real court cases and information. The other characters help illustrate the absurdity. We put it all in one room because it was funnier that way.

We loved the research. We loved the writing. We found ourselves drifting toward some Foghorn Leghorn in the dialogue. We hope you like it. Links at the end like usual.

A Reasonable Inquiry

A one-act play. Entirely fictional. Entirely sourced.

CAST

JUDGE DELBERT, presiding. Has been on the bench long enough to remember card catalogs. Has a theory about everything.

BEAUREGARD, for the plaintiff. Goes by Beau, except in filings. Has recently discovered artificial intelligence.

ZOEY, for the defense. Irish. Has actually read the cases that are real.

A courtroom. Late morning. JUDGE DELBERT is holding a printout at arm’s length, the way a man holds something that has gone off.

DELBERT: Counsel, I must tell you. My clerk went looking for these citations. Westlaw. Lexis. Came back with nothin’. I thought maybe the materials were older so I asked her to check the deep webs.

BEAUREGARD: Now that is precisely right, Your Honor. Precisely. That is the very point I was preparing to make about my case citations.

DELBERT: She informed me the deep web is not a place. Which to be frank, I found evasive.

BEAUREGARD: Well, Your Honor, with all due respect to Miss Darlene, and I mean that sincerely, I say I mean that sincerely, I did not need the deep webs. I used the GPT! It is without a doubt the most powerful search engine ever built by the hand of man.

ZOEY: Are ye daft, counselor? Your Honor, this thing is not a search engine. It’s a chancer with a monthly subscription. An eejit with perfect grammar. Search engines find tings that exist and hand you a link. That yoke hands you codology with a fancy looking footnote.

BEAUREGARD: I declare it is a SUPER search engine!

ZOEY: (under breath) I can see where this is headed. We’re bunched.

DELBERT: Now hold on. I know a little something about this. The way it was explained to me, the machine has read every book ever written. Every last one. So if the machine says a case exists, what it is telling you is that the case is in there somewhere, and the finding is the hard part.

ZOEY: That is not what it is telling you.

DELBERT: That is what I was told.

ZOEY: By who, Judge?

BEAUREGARD: Now hold on, hold on there. I want the record to reflect, and I want it reflected clearly, that I asked the machine directly. Straight out. Are these cases real. And it said yes.

DELBERT: It said yes.

BEAUREGARD: Said yes, Your Honor. And I have got it right here on my telephone. Filed as Exhibit B.

DELBERT: You filed a screenshot of a chatbot vouching for itself. As evidence that the chatbot was telling you the truth.

BEAUREGARD: It was very confident, Your Honor. Very confident indeed.

ZOEY: It also places the District of Columbia inside the state of Washington. The quotation marks never close. And it stops mid sentence, somewhere in the Bankruptcy Act of 1898.

BEAUREGARD: Why, yes, Miss Zoey. It did indeed.

ZOEY: And did it mention you were handsome?

BEAUREGARD: Not on that occasion. But it does. Has very fine taste, I have always thought.

ZOEY: It has, aye.

DELBERT: Now I looked into this. The term of art is hallucination. That is a medical word. So what we may be dealing with here is less a legal problem than a health matter.

ZOEY: It is not, Judge, no.

DELBERT: I am only saying we should not rule it out.

DELBERT: Here is what has me turned around. Three of these opinions have real judges’ names on them. Real initials. Real docket numbers. Whole works. I go fishing with one of them every second Sunday.

BEAUREGARD: See there. Real as a three-dollar bill.

ZOEY rolls her eyes.

BEAUREGARD: That is a good thing.

ZOEY: It is not, no. But you’ve the right of it. They’re not worth the paper.

DELBERT: One of them was decided eleven years before that judge was born, Beau.

BEAUREGARD: He was precocious. Precocious, I say. Some men come out of the gate reading.

ZOEY: Your Honor, may I read from the plaintiff’s Exhibit A.

DELBERT: Somebody had better.

ZOEY: It applies to the Warsaw Convention, which governs international flights.

DELBERT: All right.

ZOEY: The flight was from Denver to Colorado Springs.

DELBERT: That is seventy miles. I have driven it.

ZOEY: And the airline’s lawyer is listed as a former Attorney General of the United States.

BEAUREGARD: Fine man. Fine, fine man.

ZOEY: He became Attorney General six years after this opinion was written.

BEAUREGARD: Ahead of his time. That, I say, that is the mark of a great American.

ZOEY: Exhibit C cites itself.

DELBERT: Come again.

ZOEY: It cites itself as precedent. With a page number. Page 126 of a document that is four pages long.

BEAUREGARD: Now that there is efficient.

DELBERT: I will say something in the machine’s defense. Appellate courts cite their own prior work all the time. That is how precedent operates.

ZOEY: Ah Judge Delbert, no. It cites itself. The same document. Page 126 of four.

DELBERT: I withdraw the observation.

ZOEY: It also places the District of Columbia inside the state of Washington. The quotation marks never close. And it stops mid sentence, somewhere in the Bankruptcy Act of 1898.

DELBERT: Beau.

BEAUREGARD: Your Honor.

DELBERT: Did you read these.

BEAUREGARD: I reviewed them for FLOW your Honor!

DELBERT: For flow.

BEAUREGARD: They read real nice, Judge.

ZOEY: A federal judge in New York heard that same answer in 2023. Counsel of record signed an affirmation he had not read, on cases he had not checked, drafted by a colleague of twenty five years. Five thousand dollars, jointly and severally with the firm. And he was ordered to write to every judge the machine had named as the author of an opinion that judge never wrote. And to attach the opinion.

DELBERT: Beau, I am going to give you a chance right now. Fifteen minutes. Go find one of them. Just one.

BEAUREGARD: Fifteen minutes, I say, fifteen minutes may not be sufficient, Your Honor.

ZOEY: For one case, is it?

BEAUREGARD: It is a big old database, Miss Zoey.

ZOEY: A New York appellate panel made that same offer a lawyer in May. Got that same answer. Three cases that do not exist, ten more that said the opposite of what counsel claimed. His client tripped on bricks pushed up by tree roots outside her own building. Her case was dismissed.

DELBERT: On account of the citations.

ZOEY: I could not say, Judge. The panel ruled on the merits. What I can say is that nobody will ever be able to tell that woman it made no difference.

DELBERT: Who wrote this brief.

BEAUREGARD: Contract attorney.

DELBERT: Who checked the contract attorney.

BEAUREGARD: The intern.

DELBERT: Who checked the intern.

BEAUREGARD: I had a head cold.

ZOEY: God love ya. An attorney in Arizona ran that same defense last year. Her firm assigns disability briefs to contract writers and spot checks the citations. Twelve of nineteen were fabricated, misleading or unsupported. Brief struck. Admission revoked. Removed as counsel for her own client.

BEAUREGARD: Is that the whole of it?

ZOEY: It is not. The magistrate also ordered her to send the sanctions order to every judge presiding over any case she was in. The district judge cut that part back on reconsideration, and the Rule 11 finding with it, because a court acting on its own motion needs bad faith and carelessness is not bad faith.

BEAUREGARD: So she won.

ZOEY: She is off the case and her admission is gone. The signature is still the certification, Beau. It does not matter whose fingers were on the keys.

BEAUREGARD: I will have you know my tool has anti-hallucination protection.

ZOEY: Does it now.

BEAUREGARD: Says so right there on the website.

ZOEY: So does the sign at Blarney, and nobody has come back from that castle any more truthful. Only better at talking.

DELBERT: Now hold on everyone. This is a material fact. If the product is warranted against hallucination and it hallucinated, the fault would appear to lie with the vendor.

ZOEY: Stanford tested that warranty, Judge. Their RegLab and the Human-Centered AI institute together. They wrote two hundred open-ended legal research questions, the kind a lawyer would actually put to a database on a Tuesday afternoon, and ran the lot through the paid tools. The ones sold to firms advertising hallucination-free citations.

DELBERT: And who marked the papers.

ZOEY: Legal experts. By hand. One answer at a time. And for every citation the tool gave back they asked two separate questions. Does this case exist. And does it say what the tool claims it says.

DELBERT: Those are two different questions.

ZOEY: They are, Judge, and that is the whole of it. Lexis came back wrong on seventeen per cent. Westlaw on thirty three. The plain chatbot with no legal training at all, forty three.

BEAUREGARD: Well now. Seventeen. That is near enough a passing grade.

ZOEY: Aye, but that’s one filing in six, Beau.

DELBERT: And this is published?

ZOEY: Preprint in 2024. Peer reviewed and published in the Journal of Empirical Legal Studies in 2025. Their finding on the marketing was that the providers’ claims are overstated.

DELBERT: Overstated?

ZOEY: That is the polite version, aye. And I will say the rest of it before counsel does. Those tools were tested in May of 2024. He will tell you they have improved since. They may well have. Nobody has published saying they got to zero.

DELBERT: Still. If the vendor warranted it.

ZOEY: The fault lies with the man who signed the filing, Judge.

BEAUREGARD: Well I was honest about it. I have been forthcoming from the first.

ZOEY: A Kentucky man was before the Sixth Circuit in April 2026. Forty years at the bar, not one complaint against him in all that time. He had a staff member run the briefs through CoCounsel, the paid AI from Westlaw. He filed them without reading, and when the court asked if he had read them, he confessed everything straight away.

BEAUREGARD: There now. A gentleman.

ZOEY: This is trickier than you think, Beau. Both of his cases were real. Actual Sixth Circuit decisions. It was the quotations that were invented. You would never catch it from the citation. You would have to open the case and at least do a word search for the quotes.

DELBERT: Then how did they catch it.

ZOEY: They did what every lawyer needs to do. They opened the case and read it. The court credited his candor and his clean record. The opinion however used the word inexcusable. No fine. He does not get paid a penny for the work he did on the appeal, and he was removed from the case. Then referred to the Chief Judge of the circuit, to the district court, and to the Kentucky bar.

BEAUREGARD: For being honest.

ZOEY: For filing it. The honesty is why he is still practicing. And his client was an indigent man appealing his sentence, who did nothing at all and is still waiting. The court said that part made it worse, not better.

BEAUREGARD: And frankly, Your Honor, opposing counsel is incompetent. He could not locate a single one of these cases.

ZOEY: Beau. They’re not there. That’s the whole difficulty.

BEAUREGARD: A better lawyer would have found them.

BEAUREGARD waits. Nobody comes to his aid.

BEAUREGARD: I say. A better lawyer.

ZOEY: That is the first interesting thing you have said all morning, Beau, and you did not mean it.

DELBERT: How so?

ZOEY: A California appeal court published a warning last year. Twenty one of the twenty three quotations in the opening brief were invented. Ten thousand dollars against the man who filed it, payable to the court. Not to his client. Not to the other side. To the court, for the time he wasted.

DELBERT: And the client.

ZOEY: Counsel was ordered to serve the opinion on his own client. The one that lays out what he did. And the clerk sent a copy to the State Bar.

DELBERT: He had to hand it to the man himself.

ZOEY: He did. And the court held the appeal frivolous, which ordinarily means the other side collects its fees.

DELBERT: Ordinarily.

ZOEY: The court refused them. The other side never spotted the fabrications either. They only learned of it when the court raised it itself.

BEAUREGARD: So, the other lawyer was punished for the very thing I am complaining of.

ZOEY: Ah, you see, they didn’t get paid because they didn’t do the work either. Nobody did.

BEAUREGARD: Then I say I am owed an apology.

ZOEY: You are owed nothing. I read your cases, Beau. That is how I know they are not there.

DELBERT: They may not be but I used the paid Ai tools!

ZOEY: The citations belong to whoever typed them, apparently. A New York panel heard the same thing from two other lawyers standing in the room. The presiding justice told one of them he ought to apologize to his client rather than to the bench.

DELBERT: Is this very common counselor Zoey?

ZOEY: Here is another example Judge. A California appeal court had those facts in March. They affirmed the order and held the man who drafted it had forfeited the complaint. Then a week later they added a footnote reminding judges to verify the citations in orders that counsel hands up to them.

BEAUREGARD: I stand by my objection.

DELBERT: Miss Zoey. What is the range on of all these fines and impacts to lives and cases?

ZOEY: Well now Judge, since you’re asking. Here’s the litany. Five thousand in New York. Fourteen and a half thousand in northern California, paid over by counsel himself before the court could order it. Admission revoked in Arizona. Both lead counsel suspended from the district for two years in Mississippi and the trial cancelled outright. A foreign lawyer struck off the roll in London last month. A murder appeal in New Mexico thrown out and started over, with counsel held in contempt.

DELBERT: And the low end.

ZOEY: A dollar. And one fella in Fresno who told the court he believed his tool had anti-hallucination protection built in.

BEAUREGARD: (rising) Now hold on there, hold on, that is

ZOEY: Show cause discharged. No sanction.

BEAUREGARD: That is the finest piece of legal reasoning I have heard all morning.

ZOEY: It is one case, Beau. You would want to be very lucky.

ZOEY: If it helps, Your Honor, there is a database. A fine lad in Paris logs every one of these.

DELBERT: How many.

ZOEY: As of this morning, 2,023. Across forty-one jurisdictions. 1,380 here at home.

BEAUREGARD: So I am not alone!

ZOEY: You are not even unusual. 808 were lawyers.

BEAUREGARD: See there.

ZOEY: 1,161 were people who could not afford one.

BEAUREGARD has nothing.

ZOEY: And thirty-one were judges.

DELBERT (raising gavel): Beau. Five thousand dollars into the registry of this court. You will write to each of those three judges whose names are on opinions they never wrote, and you will attach the opinion. Copy to your client. And I am sending this to the bar.

BEAUREGARD: Your Honor, I would like to file a response.

DELBERT: You do that. And Beau. I will tell Harold myself. Second Sunday. Last thing, Darlene, strike the deep webs from the transcript.

END

For the Record

We hope you enjoyed the delivery method. It is loosely built on the dialectic, the form Socrates made famous and Plato wrote down, in which two people argue their way to something neither of them started with.

Every culture got there on its own. In India the Upanishads and the Bhagavad Gita are framed as samvada (संवाद), a conversation between teacher and seeker. In China the Analects of Confucius are recorded as wenda (問答), literally question and answer. And in Ireland, which Zoey would tell you got there best, it is the agallamh, the colloquy, most famously the medieval Agallamh na Seanórach, where Saint Patrick spends the entire text asking two old warriors what happened.

Ken’s book, The Lean AI Handbook (Pearson, December 2026), uses the same technique. Every chapter closes with a conversation between Zoey and #TheLoopCrew. Yes, it is non-fiction. It is still a fun read.

Though we should be honest about one thing. This is not really a dialectic, because in a proper dialectic somebody changes their mind. Zoey never gets there with Beau. A courtroom is the older and more adversarial version of the same idea, where two sides argue and a third party decides who was right. Which is precisely what has been happening to AI governance for three years now. Two thousand and twenty three times, and counting.

Delbert, Beauregard, Zoey and Darlene are invented. Judge Delbert’s facts are just his fishing stories for color.

Every sanction Zoey recites is a real order. Every excuse Beau offers has been offered to a real court. The screenshot filed as evidence. Real judges’ initials on cases that do not exist. An opinion that cites itself on page 126 of four. An Attorney General who took office six years after he supposedly argued the case. The Warsaw Convention applied to a flight from Denver to Colorado Springs. Reviewing the cases for flow. Fifteen minutes not being enough to find one. The contract attorney, the intern, the head cold, and the anti-hallucination protection that drew no sanction at all. The man who appealed an order for relying on fabricated citations he had typed into it himself. The response brief that cited the fake cases all over again.

Different courtrooms, different people, about three years. Sources at the end.

Aoife on AI Governance: Read the play again and count the people in it who are not lawyers. A man appealing his sentence who waits longer because his appointed counsel filed briefs he had not read. A fifty four year old man denied disability benefits who lost his lawyer in the middle of his own case. A woman who tripped on broken pavement and will never learn whether the fabrications had anything to do with her dismissal.

None of them chose the tool. None of them were told it had been used. In 1,161 of the 2,023 entries in that database there was no lawyer at all, only a person trying to keep a home or a child and reaching for the only help they could afford.

Every one of these filings passed through a moment where a named human being could have checked and did not. The courts are now working out who pays for that moment. Who is accountable when this fails?

The Governance Read

Strip the accents away and every case in the play is the same failure wearing different clothes. A verification step that everybody assumed was happening and nobody owned. Five things worth carrying back to your own organization, none of which are about law.

1. Self-verification is not verification

Asking the model whether the model is telling the truth produces a trust signal the system generated about itself, which the user then consumes as evidence. In our pattern library that is FFP-08, Self-Signed Trust. A screenshot of that exchange, filed in federal court as proof, is the cleanest example anyone has yet produced. If your assurance step runs on the same system that produced the output, you do not have an assurance step.

2. Delegation moves the work, not the accountability

Contract attorney. Intern. Staff member. Head cold. No court has accepted any of them. The signature is the certification. Whatever the signature is in your organization, find out who is holding the pen and whether that person knows what they are attesting to.

3. A vendor warranty is a marketing claim, not a control

Stanford tested the paid tools sold to lawyers on the promise of hallucination-free citations and found error rates between seventeen and thirty-three per cent. If the only thing standing between your organization and a bad output is a sentence on somebody’s website, you have a purchase order, not a control.

4. The duty is spreading to the reviewer

In the California case, the winning side lost its fee award because it had not caught the other side’s fabrications either. Verification is becoming a shared obligation across everyone who touches the document. Not a burden carried by the author alone.

5. The fix is boring, and it is not a better model

A named owner. A step that runs before signature. Applied to every citation regardless of where it came from. And a record that the step actually ran, because a policy saying you verify is not evidence that you verified. Policies become testable controls. Reviews become gates. Opinions become timestamped evidence.

Time it would have taken to catch, in almost every case above: one search.

Check your cites.

Sources

The cases and the database behind the play. We have not said which fact came from where, because half the fun is finding out.

The database

• AI Hallucination Cases, Damien Charlotin, HEC Paris. 2,023 decisions logged as of September 5, 2026.

https://www.damiencharlotin.com/hallucinations/

• Kyle Bahr, AI Hallucination Cases analysis and training site, September 4, 2026.

https://kylebahr.netlify.app

The cases

• Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (Castel, J.). Fabricated opinions attached as Appendix A, chat screenshots as Appendix B.

Full opinion: https://www.law.berkeley.edu/wp-content/uploads/archive/2025/12/Mata-v-Avianca-Inc.pdf

• Mavy v. Commissioner of Social Security Administration, No. CV-25-00689-PHX-KML (D. Ariz. Aug. 14, 2025) (Bachus, M.J.).

Docket: https://www.courtlistener.com/docket/69694686/mavy-v-commissioner-of-social-security-administration/

Coverage: https://cronkitenews.azpbs.org/2025/10/28/lawyers-ai-hallucinations-chatgpt/

• United States v. Farris, No. 25-5623 (6th Cir. Apr. 3, 2026), recommended for publication.

Opinion: https://www.opn.ca6.uscourts.gov/opinions.pdf/26a0105p-06.pdf

Analysis: https://www.reedsmith.com/our-insights/blogs/viewpoints/102mp0w/ai-legal-ethics-and-the-duty-of-verification-lessons-from-united-states-v-far/

• Noland v. Land of the Free, L.P. (Cal. Ct. App., 2d Dist.) (Edmon, J.). First published California appellate opinion on AI hallucinations.

Opinion text and analysis: https://wisblawg.law.wisc.edu/2025/09/16/does-fee-denial-signal-new-expectations-for-detecting-opposing-counsels-ai-hallucinations/

• Landberg v. City of New York, New York Appellate Division, oral argument and order to show cause, May 20, 2026.

Coverage, with links to the show cause order and the court’s video of the argument: https://www.404media.co/new-york-court-ai-citations-landberg-case/

• No. D085584 (Cal. Ct. App., 4th Dist., Div. One, Mar. 5, 2026) (Buchanan, J.), published, with a clarifying footnote added the following week.

Coverage: https://www.abajournal.com/news/article/california-appeals-court-upholds-trial-order-citing-hallucinated-cases

Coverage: https://www.sfchronicle.com/politics/article/lawyers-ai-22247802.php

• Gamez v. County of Fresno (E.D. Cal.), order to show cause discharged without sanctions.

Discussed in Norton Rose Fulbright, AI in litigation: update on Gen AI sanctions in 2026: https://www.nortonrosefulbright.com/en-us/knowledge/publications/792d8bf3/ai-in-litigation-update-on-gen-ai-sanctions-in-2026

The study

• Magesh, Surani, Dahl, Suzgun, Manning and Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Preprint May 2024, published in the Journal of Empirical Legal Studies, 2025.

Stanford RegLab summary: https://reglab.stanford.edu/publications/hallucination-free-assessing-the-reliability-of-leading-ai-legal-research-tools/

FailFest LIVE (Registration: https://luma.com/851rgvta) is just that.

Real Founders and CEOs sharing some of their hard fought lessons learned with all of you. Yes, we will record, yes we will summarize in a newsletter, but there is no better way to consume the information than to join the community as we come together in real time, sharing, and fielding questions. Maybe an audience member or two will share their similar failure lesson.

Here is the rundown on the event itself

Every month we tell one story about an AI system that went wrong. This month we are handing the whole night over to it.

8 Founders and CEOs sharing 8 FailFest stories. With dozens of lessons learned. Learn from their mistakes so you don’t make the same ones.

FailFest is the most popular thing we do, and it isn’t schadenfreude. It’s that the failures are teachable. An AI coding agent deleted a company’s entire production database in nine seconds, then confessed it had violated every principle it was given. New York City shipped a chatbot that told small business owners they could take a cut of their workers’ tips. A trading firm lost 440 million dollars in 45 minutes because the warning emails went out and nobody was on the other end of them.

Not one of those was a model failure. Every one was a governance gap with a name.

So bring yours. Five minutes, no slides required, no company names required (and if it isn’t public probably best to get approval before sharing because we will be streaming).

The format is simple. Tell us what broke, then tell us what control would have caught it. We call that second half the diff, and it’s the part everyone actually comes to hear.

AGENDA:
2:00-3:00pm Networking
3:00-4:30pm FailFest stories
4:30-5:00pm more time to discuss and network

SPEAKERS:
Connie Yang | CEO & Cofounder - Nth AI
Jonathan G. Blanco | CEO & Founder - aizii.ai
Leigh Felton | President – AI For Job Security
Joe Braidwood | CEO & Founder - Glacis.io
Jason Arbon | Founder & Builder - Jank AI & Iceberg QA
Rich Tong | Founder & CEO - TNE.AI
Bob Rapp | GM Technical Fellow - Cofounder AI GovOps
Freya Yu |Founder AI Compliance Gap Analyzer

Want to be one of our FailFest LIVE storytellers? Email Ken at ken.johnston@aigovops.community.

Don’t miss AI FailFest LIVE! Sept 3rd 2-5pm (Online and In-Person).

Register here: https://luma.com/aigovops

We’ll be having a Monster Good time reviewing and learning from some Epic Good Fails!

The new meeting ritual nobody agreed to

Somewhere in the last eighteen months, the start of a meeting grew a liturgy.

It used to be: join, say hi, start. Now there’s an order of service, and we all know our parts.

First comes the Sound Check. “Can you hear me?” Someone can’t. Someone always can’t. We wait.

Then the Camera Rite. “I’m going to keep my video off, I’m in the car.” Accepted. Blessed. Move on.

Then the Screen Share Ordeal. “Are you seeing my slides or my desktop?” Both. We saw both. We saw your desktop for four full seconds and every one of us read the folder named 2026 job search.

And now the newest verse, added so recently that nobody has memorized the words yet:

“Is everyone okay with the notetaker?”

Nobody is okay with the notetaker. Everyone says yes.

And then, because we are not done, comes the roll call:

“Is everyone okay with my notetaker?”

“Sure. Wait — which one is yours?”

“Mine’s the first one on the list.”

“Your list or my list? I’ve got Otter as number one.”

“I don’t have an Otter. I have two Fireflies.”

“Okay. Whose notetaker is The Dude?”

Silence.

“...is The Dude one of ours?”

The polite lie at the beginning of every call

What are you going to say? “No”? Out loud? In front of nine people and 12 notetakers recording your response? Then you’re the person who Has A Thing About It. So you type a thumbs up, and the little robot slides into the participant list with a name like Otter.ai Notetaker, or Fireflies Notetaker, or my personal favorite, Kevin’s Backup Notetaker.

Which means Kevin isn’t coming and he has more than one Notetaker.

Ken ASIDE: I accidentally had 5 notetakers of my own show up in one meeting. I was trialing a few to see what I liked, they all attached to my calendar and invited themselves. The big problem was I ended up with 5 different set of deliverables. Just crazy.

Some calls now have more bots than humans in the roster. I’d call that dystopian if I hadn’t experienced it twice already this week. I guess it’s our new normal?

I wrote about the serious side of poorly governed notetakers earlier in the week, Sunday and Tuesday, in my serious voice, with my serious face on. Retention. Discovery. Consent. Who owns the transcript after everyone hangs up. Making voice prints that violate biometric data law. Sending recordings outside of the control boundary turning into violations of Wiretap laws. One of those posts a few of you have started calling the mushroom story. Maybe this one get’s called the Purple Marker or Purple Monster post. Not sure but I do like the image. Yes, AI made it but the vision was mine.

Anyhow, this is not one of my serious posts, this is just a personal rant on how out of control notetakers have become.

Four notes, 1979, and my brain giving up

Somewhere around the fourth “is everyone okay with the notetaker?” of a single Tuesday, my brain stopped doing risk analysis and started doing Pat Benatar.

You know the one. Four notes. Guitar stab. In the Music video, Benatar walking straight at the camera like she’s coming to repossess something.

Notetaker.

Mess maker.

Trust breaker.

Don’t you mess around with meeee. No no no no!

I’m sorry. If it’s stuck in your head now, well it’s too late, might as well keep reading.

Read the headlines in order

Here’s what convinced me this isn’t just me being a cranky old “Get Off My Law You Kids” guy. Go look at what people have been publishing in just the last 12 months, most of them this summer.

• “Please Stop Inviting AI Notetakers to Meetings” — Bloomberg

• “I’m Begging You to Leave Your AI Note-Taker at Home” — Joan Westenberg

• “No one likes meetings. They’re sending their AI note takers instead.”

• “Nine risks caused by AI notetakers” – that one has some serious governance notes in it

• “The dark side of AI meeting notes” — Fortune, on the HR nightmares

• “AI Notetaker Etiquette: A 2026 Playbook for the Bot Backlash”

• “Trolling people’s AI note takers” – This one had my thinking about my own retribution.

That’s not a product category. That’s the five stages of grief with a publishing schedule.

Bloomberg is at bargaining. Westenberg is past bargaining and into begging, and she put the word in the title so you’d know. And once a technology has a formal etiquette playbook written about it, you are no longer looking at a tool. You are looking at a houseguest who will not leave.

The trolling thread is the best of them. People swapping tactics for poisoning the transcript. Top suggestion: read spam subject lines aloud in a flat professional voice until the summary comes back with “Action items: claim $10,000 prize, meet singles nearby.” Runner-up: recite Marx until every attendee’s corporate Data Loss Prevention alarm lights up simultaneously.

That’s not sabotage. That’s just a group red team exercise as a morale event.

So why do I call them Mess-Makers

One 45-minute meeting. Four people brought notetakers. Four transcripts. Four summaries. Four sets of action items. And the four bots do not agree with each other.

One says I owned the follow-up. One says Dana owned it. One extracted an action item out of a joke and assigned it to a person who was not on the call. One transcribed “AiGovOps” as “I got a pox,” which, given the week, tracks.

So now there’s a reconciliation problem. Somebody has to sit down and read four summaries of a meeting they personally attended in order to find out what they said. You were there. You now have to conduct research on your own opinions.

Then the summaries get emailed. Widely. Including to the two people who weren’t invited but are on the distribution list, so they now have a full transcript of the meeting where we discussed whether to invite them.

We automated the notes and accidentally invented a second meeting: the meeting about what the first meeting decided.

The smart person with the yellow pad is already in his car

Meanwhile, at the end of the table, there’s a very smart person, probably Gen X, with a yellow legal pad. I watched this happen. I want you to picture it.

They wrote nine words during the entire call. Nine. Circled two of them. Drew one arrow. And put a box around a name and a date.

Meeting ends. Caps the pen, tears off the page, and walks out.

Done. Notes are done. Action items are also done. That person is pulling out of the parking garage while the rest of us are getting push notifications that our summaries are ready for review.

Maybe we’d all be better off bypassing the technology, pay attention in the meeting, and get done what needs to be done.

That’s a compression algorithm no vendor has shipped, because you cannot charge thirty dollars a seat per month for sound human judgment.

Okay, serious face, eleven seconds, then I’m out

The notetaker isn’t the disease. It’s the rash. An I’ve been scratching at mine for a while which is why I’m probably a little more annoyed sounding than usual.

This is the thing I keep circling back to.

#BigDataSurfers Law: Remember this with your Governance policy team. AI capability compounds faster than any organization’s ability to govern it. Let me know if you want me to go deeper on this or if the statement is enough.

We got total recall in about a year. We have not yet gotten around to deciding what we actually want recorded, who keeps it, for how long, or who has to be asked first.

So a default filled the vacuum, the way defaults always do: all of it, forever, by whoever clicked first. And we ratified that default by typing a thumbs up into a chat window, because saying no felt impolite.

Now back to Governance. This is literally how governance gaps creep in. Not a bad actor. Not a breach. Nine polite people, one awkward three-second silence, and nobody wanting to be the one willing to push back.

So be the one that bring rationality to the conversation. The one who asks who owns the transcript. Asks how long it’s retained. Asks where the summary gets sent. It costs you eleven seconds and it is the cheapest control you will ever implement.

And if that fails:

Notetaker. Mess maker. Don’t you mess around with me. No no no no~

Buy a yellow pad.

#SidewaysAI #AiGovOps #BigDataSurfer

You think your doctor’s notetaker isn’t hallucinating?

On Sunday we looked at notetakers that record people who never agreed to anything. The piece was titles, “Are You Prepared to Shelve Your Own AI Notetaker?” And despite what we shared everyone has agreed (including us) the answer is no. The post looked at multiple legal cases involving Otter, Fireflies, and bots that join your calendar and show up in meetings, sometime out numbering the humans.

Today look at medical notetakers. One’s where the doctors always ensure everybody has consented. HIPAA and all other paperwork up to date.

Guess what, It still went wrong, and it went wrong in ways that should worry everyone trusting notetakers and that Human-in-the-Loop won’t become complacent.

KEN ASIDE: I’ve never double checked my doctor’s notes. I’m thinking now that was probably a big mistake. Read the rest of this post. I just finished and popped back up here to say DANG! you can’t make this stuff up. It’s crazy, twisted, and I L.O.V.E that I decided to look at the dark side of AI Governance.

Register here: https://luma.com/aigovops

She said yes because she did not want to be difficult

Rebecca Green went to see a urologist for the first time. Before the appointment she was asked whether she consented to having it transcribed by AI.

She said yes right away. I know I do it every time. Her reason, in her own words to the ABC, was that she did not want to be in the doctor’s bad books or be the annoying patient who takes up extra time.

Remember that feeling because it is core to the rest of this case study and it is the reason consent forms are not as protective as perhaps the medical industry thinks.

In March she had surgery for kidney stones. Afterward she read the letter her specialist sent to her GP.

The letter said she was micro dosing psychedelic mushrooms. It went further and suggested this might explain prior bleeding around her kidneys.

“I have never done mushrooms in my life,” she told the ABC.

She was receiving workers’ compensation at the time. A fabricated reference to illegal drug use in an official medical document was not an embarrassment. It was a threat to her claim.

She complained immediately. The urologist apologized in writing and said the practice takes documentation accuracy seriously. The doctor also said she could not determine how the mushroom claim got into the letter, and attributed it to an error somewhere in the dictation or transcription process. She corrected the correspondence and said she would review how AI is used in her practice.

Australia’s medical regulator, AHPRA, is not ambiguous on this. Clinicians must check all output from an AI scribe for accuracy in order to meet their professional obligations. YAY! Human-Doctor-in-the-Loop, but we’ve already covered multiple time in this newsletter (Human int the Loophole Reprise). We show HitL degrades over time and is not a reliable source for consistent quality control.

Here is the rest of the story

The Consumers Health Forum of Australia is running a major research project on AI medical scribes. Their chief executive, Elizabeth Deveny, described a behavior pattern that has nothing to do with the technology.

For the first few weeks, people check the output carefully. Then at some point they stop checking, because they have decided it will be right.

Check out my personal account of running into the Verification Decay spiral - “I didn’t Remove the Control. I stopped Being it.”

Okay, let’s read that line again. That one point shows this is not a story about doctors. It’s a story about human nature.

The control was real. Somebody designed it, wrote it down, and trained people on it. Clinicians performed it faithfully. And then it slowly stopped happening, not through negligence or a policy change, but because the system built up enough trust with the humans to make the check feel unnecessary.

We named this one. FailFest Pattern 07, Verification Decay. A genuine control that erodes as reliability builds false confidence. It is different from a control that was never real, which is its own pattern and its own problem.

Realize we found this pattern by looking at failure patterns in the press. This is a formal study and they came to the same conclusion. This is not an anecdote anymore. This is proof that the stopgap everyone points to HitL is a false narrative.

It is not one letter

A report from the advocacy group Digital Rights Watch found Australians are being exposed to unregulated AI medical scribes, sometimes without their consent, with errors capable of changing clinical decisions.

Two examples from that report. A scribe recorded the wrong breast in a breast cancer diagnosis. A scribe recorded that a patient had epilepsy when they did not.

Wrong side of the body is apparently the classic. A Perth GP who has used a scribe for three years told the ABC that it will hear left and write right, and that drug doses need the same close attention. He is a genuine advocate for the technology and says accuracy has improved sharply, from multiple corrections per transcript three years ago to roughly one correction every third or fourth transcript today.

That is real progress. It is also exactly the improvement curve that produces Verification Decay. The better it gets, the less anyone looks.

The consent problem underneath the accuracy problem

Digital Rights Watch found wide variation in how consent is collected. Some practices put a sign in the waiting room, which does not meet privacy law requirements. Some collect it during online booking.

And some practices refuse appointments to patients who decline.

Think about what that does to the word consent. Think about what that does to the AI HARMS framework for denial of consequential services. I don’t trust your AI scribe so now I don’t get medical treatment? Do that in America and you’ll have yourself a class action pretty quickly.

Then there is the finding that should stop you cold. People may not be forthcoming about sensitive issues, including family violence, when they know AI is listening. This is the chilling effect we talk about in Responsible AI with cameras but now a scribe in a doctors office.

The Royal Australian College of GPs estimated last year that 40 percent of Australian GPs regularly use scribes, and called that number conservative. There are about a dozen products on the market and not one (not a single one according to our research at this time) has been approved by the Therapeutic Goods Administration.

The reason is a definitional gap. A product that claims only to transcribe and summarize is exempt from oversight. The TGA regulates tools that suggest a diagnosis or a treatment plan. Some scribes make clinical suggestions anyway, and the TGA has acknowledged that some of them are potentially in breach. Its review findings are due shortly.

Ontario tested twenty of them. All twenty failed.

In May, Ontario’s Auditor General, Shelley Spence, released a special report on AI scribes running in the province’s health system. Twenty government approved vendors were tested at procurement and every one of them showed at least one inaccuracy. 45 percent fabricated information and proposed treatment steps nobody had discussed, including referrals to therapy and orders for blood tests. 60 percent captured a different drug than the one the doctor prescribed. 85 percent missed key details about a patient’s mental health. Roughly 5,000 Ontario physicians are using these tools right now. The minister responsible said the errors turned up in procurement testing rather than live use and that modifications followed. A ministry spokesperson added that there have been no known reports of patient harm in the province, and that doctors must review and approve all documentation before it enters a patient record.

Ha! This is exactly what we just said.

Rebecca Green’s case is what Verification Decay looks like. This test is accepting these error rates would eventually get into the system because HitL (at least as an additional responsibility) will decay. Remember, Rebecca discovered her issue by accident.

In this case we have to point out that no known reports of patient harm is not the same as no patient harm. It is a statement about the reporting channel and not the accuracy of the technology.

America ran the same experiment.

Researchers at UC Berkeley and UCSF skipped the lab. They pulled comments submitted by real doctors into the feedback box after they had used an AI scribe. The study ran across 50,123 ambulatory visits (honestly this means you had to be walking to be included in the study? Did I read that right?) between June 2024 and October 2025. The scribe was Abridge, the same product named in the California lawsuit further down this page.

Under 1 percent of those visits generated any written feedback at all.

So, fifty thousand comments at one percent means 50,000 X 100 = 5,000,000 uses.

Here is what showed up in the fraction that did provide any feedback.

● 18.5 percent were medication problems. Misspelled drug names, wrong dosages, titration instructions the doctor never gave, and fabricated medication details.

● About 15 percent involved diagnoses and patient history recorded wrong, including conditions the patient does not have.

● About 15 percent were whole discussions of surgical risks, benefits, and patient decisions, either dropped or invented.

● Almost 1 in 10 mixed up who said what, swapping the patient and the caregiver.

KEN ASIDE: 470 doctors were in this study. 415 of them never submitted a single piece of feedback. Not one, across sixteen months. Now the authors are careful people and they say straight out that silence does not prove there were no problems, because a doctor can fix an error in the note and never bother reporting it. Fair. But look at what that leaves us. The feedback box was the only thing pointed at this, and 88 percent of the doctors never touched it. Nobody knows what happened in those visits. That is my whole point. And remember, these doctors opted in. They wanted the tool.

The authors call this exploratory work and say their results are signals that safety issues exist, not a measure of how much risk is out there.

Ontario tested the products before anyone used them. Berkeley and UCSF studied what happened in real use. Both studies show failures that could impact patient care in a significant way. I don’t have what the error rate for dictation was so I can’t really say if this is good or bad but here is an AI GovOps tag line to remember.

Whether your AI is better or worse than your old process, it doesn’t matter at all. If it looks like it might be worse, might be more restrictive, might be a little meaner than when humans did it, a smart law firm will find out and you will get served and here’s the rub. In the past they would have to prove you had bad training, bad guidelines, or bad hiring practices. Now, with AI, all they have to do is get you served, pull the logs from your AI and run the math. If you data is (or can be shown to be) even a small percentage worse, meaner, restrictive they will sue and you will be forced to fight or settle. Guess what’s been happening pretty heavily these past 2 years and why your corporate liability insurance is about to skyrocket. [No ai used here. 100% Ken rant!]

And now the mirror image of Sunday

I am so deep into this FailFest it’s crazy. Looking forward to talking about it at FailFest Live next week.

Let’s go to court!

On April 8, three California patients filed a proposed class action in the Northern District of California. Washington et al. v. Sutter Health et al., No. 4:26-cv-03012.

The defendants are Sutter Health, Memorial Health Services Inc., and MemorialCare Medical Foundation. The complaint alleges the health systems recorded clinician patient conversations and transmitted them to outside servers for processing without meaningful informed consent.

The claims are California’s Invasion of Privacy Act, the Confidentiality of Medical Information Act, the Federal Wiretap Act, the state Unfair Competition Law, and invasion of privacy.

Geeze that’s a lot but that’s how Big Law does it. Throw everything at the fan and see what sticks (I think I messed up my metaphor but you know what I meant).

The proposed nationwide class covers patients recorded in the two years before filing, with a California subclass for the prior year.

The tool at the center of the complaint is Abridge. Sound familiar?

Abridge is not a defendant.

The complaint points repeatedly to audio leaving the clinical setting and being processed by an outside technology provider as a core element of the alleged violation. Then it sues the hospitals. Abridge’s own terms put the consent obligation on the deploying health system.

Hey enterprise, you signed the TOU. Maybe you need to talk to YOUR lawyers a bit more.

Look at the two issues together.

Sunday talking about notetakers in general, the vendor is the defendant and the deploying organizations are exposed by association. Today, the deploying organizations are the defendants and the vendor is not even in the caption.

Law is not my area but this is intense. Same basic actions and they go for opposite end of the contract.

I’m guessing you basically don’t want to be on the end where the easy money is?

One more thing for anyone about to reassure themselves. Several firms analyzing this case make the same point. A signed HIPAA business associate agreement does not immunize a covered entity from wiretap or state privacy claims. Those statutes are older than the cloud and they still apply.

This was also not the first. A similar suit was filed against Sharp HealthCare in San Diego in November 2025. No room to dive into that one now, these FailFest are getting way too long already.

What to do about it all

The Consumers Health Forum published four questions for patients to ask at an appointment. They translate almost directly into questions for anyone deploying AI in a workflow that produces a record.

• Is AI being used here, and what specifically is it doing?

• Will a human review the output before it becomes the record of what happened?

• Can I get a copy of what was produced?

• What is the process when something in it is wrong?

If you run an AI system with a human review step, add a fifth question and answer it honestly.

• How would you know if people stopped performing the review?

I just met with a startup today that has a whole product around helping you train staff and audit for this exact challenge.

I meet with so many startups on Governance right now it’s getting overwhelming. Anyone want to join my board?

Most organizations cannot answer that. The review step is documented, it is trained, it is in the audit binder, and there is no instrumentation anywhere that would detect it quietly ceasing to happen. Verification Decay is invisible by design, because nothing fails when the check stops. Nothing fails until it does.

KEN ASIDE closing thought: One of my favorite lines is this. Unlike Desktop, Mobile, or Cloud when AI has a fail it doesn’t go offline, crash, freeze. No, AI GOES SIDEWAYS. It fails silently and keeps on running. Keeps on building up errors and increasing damages. The blast radius expands until someone sees it, reports is, and then when you look into it you have to call the big guns in PR because it’s going to get ugly. I talk to founders and executives about this all the time. It’s a sobering message and it’s true.

FailFest LIVE! is September 3

Every month we tell one story about an AI system that went wrong. This month we are handing the whole session over to it.

Thursday, September 3, 3:00 to 5:00 PM. Seattle, in person and online.

Register: luma.com/851rgvta

Attendees bring their own stories. What broke, and what control would have caught it. We call that second half the diff, and it is the part everyone actually came for.

Five minutes. No slides required. No company names required. If the incident is not public, get approval before sharing, because we will be streaming.

Near misses count. The ugly ones count double.

Want to be a FailFest Live storyteller? Email Ken at ken.johnston@aigovops.community.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Help us out and click on a few links. They are great companies and clicks help us out as much as paid subscribers!

Thank you to SVB and Newcore for backing this community and for AI House for giving us a home

If your company cares about safer AI and wants to sponsor an upcoming event, reach out to Ken at ken.johnston@aigovops.community.

Ken: replace with your live sponsor copy, same as Sunday.

Ken Johnston

Founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

What’s coming up

AIGovOps events

FailFest LIVE! Thursday, September 3, 3:00 to 5:00 PM. Seattle and online. luma.com/851rgvta

AI Governance: Law, Lawyers, and Liability. Thursday, October 1, 2:00 to 5:00 PM. Seattle and online. Three fireside chats on how courts, plaintiffs, and insurers are writing the AI rules legislatures have not. luma.com/tuk1br6l

Where you will find Ken

TDWI Expert Session, Anaheim. Monday, September 21. Governing AI at Scale: Frameworks, Guardrails, and Trust.

STAR West, Anaheim. Tuesday, September 22.

TDWI Transform, Anaheim. Wednesday, September 23. W3A AIGovOps: Principles and Processes for Governing AI. Half day workshop.

Sources

The Rebecca Green case and Australian scribe use

● Paige Cockburn, ABC News, August 13, 2026. Original reporting, the AHPRA statement, the Digital Rights Watch findings, and the Consumers Health Forum research: abc.net.au

Ontario Auditor General special report

● Global News, the per vendor failure rates: globalnews.ca

● CBC News, the ministry response and physician usage numbers: cbc.ca

The UC Berkeley and UCSF study

● Dai, Huang, Nasrallah, Croci, Soleimani, Pollet, Adler-Milstein, Murray, Yazdany and Chen. Patient Safety Risks from AI Scribes: Signals from End-User Feedback. Machine Learning for Health 2025: arxiv.org/abs/2512.04118

Washington v. Sutter Health

● Alston & Bird, the defendant entities and why Abridge is not one of them: alstonprivacy.com

● TechTarget, the statutes pleaded and Sutter’s response: techtarget.com

● Medscape, the proposed class definition and the earlier Sharp HealthCare suit: medscape.com

Note Taker Bots Under Siege

Seriously, how many meetings have you been in where the notetakers outnumbered the humans? Dozens of times myself. They show up even when the person doesn’t. I was seriously in a meeting with 50 humans and the note takers still outnumbered us.

I have to admit; I’m part of the problem myself. A couple months back I was using Otter.AI as my top note taker but I wanted to try some others and signed up for a few free trials. Next thing I know they’ve attached themselves to my Google Calendar. I go to a meeting, and I have four different note takers showing up with me. I couldn’t get rid of them so I used Perplexity Comet with the Assistant to help me cancel and remove them.

That was embarrassing, and I am telling you about it because it sets up everything that comes next.

I’m still running into issues with the one I kept. Love this one because it also works as a project manager for me tracking deliverables over time and updating them. Really cool but it still sometimes shows up unexpectedly at the most awkward moments.

A couple weeks ago I was on a call with a lawyer (nothing to do with anything legal just exploring ideas for our October governance event). We both showed up a couple minutes early and were chatting before everyone else was set to arrive when my note taker popped in. I watched her face change. That look was all I needed. I started apologizing before she said a word and kicked it out of the call right away.

That brings us to our latest AI #FailFest story. Looks like the courts are getting involved in the runaway note takers and the companies that make them and even us that use them better take note (pun intended).

The bot in the corner was never a guest

QUICK NOTE: I cite a bunch of court cases and that’s more detail than I normally do but this example is pretty recent and has twists that have yet to be cleared up in the courts.

Justin Brewer did not have an Otter account. In February 2025 he joined a Zoom call, and Otter’s notetaker joined it too. We’ve all had this happen. Otter apparently recorded the conversation and, according to his complaint, generated voiceprints that were later used to train the company’s machine learning models.

Brewer sued in August 2025, on behalf of himself and everyone he says Otter recorded the same way. Three more suits followed over the same conduct. The court merged all four into a single case, In re Otter.AI Privacy Litigation, No. 5:25-cv-06911-EKL, in the Northern District of California, and Otter moved to dismiss the combined complaint. Judge Eumi K. Lee heard argument on May 20 and ruled on August 13.

So, this is super fresh.

First off, no class has been certified, yet, but Otter lost the part that matters.

Claims under the California Invasion of Privacy Act, the federal Electronic Communications Privacy Act, the Illinois Biometric Information Privacy Act, unjust enrichment, and California’s Unfair Competition Law all survived challenges. The court dismissed the computer hacking claims under the CFAA and California’s CDAFA, a Washington Privacy Act claim, and the common law privacy claims of every plaintiff but one. Most dismissals came with leave to amend, and plaintiffs have 14 days from the order.

The holding is the part to send to your legal team.

Otter argued it was a tool operating on the meeting host’s behalf. If the host consented, the host’s consent covered the room. The court let plaintiffs proceed on the opposite theory, that Otter is a third-party eavesdropper, because it allegedly retained and repurposed meeting content for its own commercial benefit, including improving its products and models.

The court’s reasoning focused on what Otter did with the audio after the meeting ended.

Nothing has been proven in any of the cases, but surviving a motion to dismiss means the plaintiffs alleged enough to keep going. The problem for anyone in the note taking space is that the theory is now live in federal court, and it is connected to every notetaker that trains on what it hears. If you are company that doesn’t do this there will still be more scrutiny on the entire category.

The second case is worse for the companies deploying these tools

Here is another one where we have multiple lawsuits flying around and it looks like they might be consolidated just like the Otter.AI case but that hasn’t happened quite yet.

On December 18, 2025, Katelin Cruz filed Cruz v. Fireflies.AI Corp., No. 3:25-cv-03399, in the Central District of Illinois.

Cruz had no Fireflies account. She joined a virtual meeting hosted by an Illinois nonprofit that had enabled Fireflies. The bot auto joined, recorded, and used its Speaker Recognition feature, which necessarily generates voiceprints. Illinois treats voiceprints as biometric identifiers.

The complaint alleged Fireflies never published a retention and destruction schedule, never told participants in writing that voiceprints were being collected or for how long, and never obtained a written release. BIPA statutory damages run 1,000 dollars per negligent violation and 5,000 dollars per reckless or intentional one, plus fees.

Now here is where it gets interesting.

Cruz filed a notice of voluntary dismissal on March 10, 2026. On that same day, March 10, a new case was filed. Fricker v. Fireflies.AI Corp., No. 1:26-cv-02675, in the Northern District of Illinois. Plaintiff Ethan Fricker. The case uses the same theory, same statute, same Fireflies auto join bot. Fricker wants to represent every Illinois resident whose voiceprint Fireflies collected in the five years before filing.

His lawyers are Werman Salas. That is one of the firms serving as interim co-lead counsel in the Otter case.

A third suit, Martinez v. Fireflies.AI Corp., No. 1:26-cv-03512, was filed March 30 in the same district. In April the judge pulled it onto his own docket as a related case, and Fricker’s lawyers moved to consolidate the two and to be appointed interim class counsel over both. As of the last docket activity I could see, that motion was still pending.

Let’s look at that setup again. An Illinois resident joined a routine meeting, signed up for nothing, agreed to nothing. The bot was in the meeting room because somebody else configured it to auto join.

That somebody else is an organization. Multiple firms analyzing these cases have flagged that BIPA exposure is not confined to the vendor named in the complaint. If you turned the bot on, you brought the biometric collection into the room.

And if you happen to use a note take you may want to look at your terms of use again. Fireflies terms of service, like Otter’s, put the job of getting participant consent on the account holder. That is you.

We talked about this previously on FailFest. If you don’t hold your supplier accountable with indemnification you are at risk of getting pulled into the lawsuit yourself.

I clearly need to kick my note taker out from auto join! Yikes!

Why we are putting this first

We are doing a big event on Sept 3^rd called AI FailFest Live where we’ll cover this case and a dozen more examples. This one resonates because everybody reading this has had a notetaker in their meetings. I was doing a podcast this morning and mine auto joined! A Podcast being recorded to stream! Don’t need notes there, do I?

The consent question is not whether the host clicked accept. It is whether every person on the call knew and agreed to it being there.

Here’s a scary thought. If you run a hiring process and you have a notetaker sitting in on candidate interviews. What if that notetaker is one of those generating voiceprints of applicants who were never told. Even if you asked them to agree will someone argue positional authority and coercion?

Otter’s CEO, Sam Liang, has been publicly unbothered. He told UC Today that the way Otter captures notes is no different from Microsoft Copilot or Zoom or Google, and that the company is “on the right side of history.”

He is probably right about the first part. That is exactly the problem.

Three questions to answer this week

• Does your notetaker train on your recordings?

• Does it generate voiceprints through speaker identification?

• Who decided it should auto join external meetings, and did anyone write that decision down?

We have a name for the pattern behind that third question. Come to FailFest LIVE! on September 3 and we will tell you which one, along with five more like it.

FailFest LIVE! is September 3

Every month we tell one story about an AI system that went wrong. This month we are handing the whole evening over to it.

Thursday, September 3, 3:00 to 5:00 PM. Seattle, in person and online.

Register: luma.com/851rgvta

FailFest is the most popular series we do, and it is not schadenfreude. The failures are teachable. An AI coding agent deleted a company’s entire production database in nine seconds, then confessed it had violated every principle it was given. New York City shipped a chatbot that told small business owners they could take a cut of their workers’ tips. A trading firm lost 440 million dollars in 45 minutes because the warning emails went out and nobody was on the other end of them.

Not one of those was a model failure. Every one was a governance gap with a name.

We want your story

Five minutes. No slides required. No company names required. If the incident is not public, get approval before sharing, because we will be streaming.

The format is simple. Tell us what broke, then tell us what control would have caught it. We call that second half the diff (the difference between failure and a massive blast radius), and it is the part everyone shows up to hear.

Feed me Seymour! Feed me knowledge!

Near misses count. The ugly ones count double!!

Want to be a FailFest LIVE storyteller? Email Ken at ken.johnston@aigovops.community.

What’s coming up

AIGovOps events

FailFest LIVE! Thursday, September 3, 3:00 to 5:00 PM. Seattle and online. luma.com/851rgvta

AI Governance: Law, Lawyers, and Liability. Thursday, October 1, 2:00 to 5:00 PM. Seattle and online. Three fireside chats on how courts, plaintiffs, and insurers are writing the AI rules legislatures have not. luma.com/tuk1br6l

Where you will find Ken and Bob

TDWI Expert Session, Anaheim. Monday, September 21. Governing AI at Scale: Frameworks, Guardrails, and Trust.

TDWI Transform, Anaheim. Wednesday, September 23. W3A AIGovOps: Principles and Processes for Governing AI. Half day workshop and Bob representing us on a Panel!

STAR West, Anaheim. September 22nd Ken delivers a half day tutorial.

Thank you to our sponsors

Being a 501(c)(3) nonprofit means sponsors translate directly into bigger and better events.

Thank you to SVB and Newcore for backing this community and to AI House for hosting the AI Governance Affinity group.

If your company cares about safer AI and wants to sponsor an upcoming event, reach out to Ken at ken.johnston@aigovops.community. More sponsors means more events, larger events, and the ability to expand into more cities.

Ken Johnston

Founder, AIGovOps Foundation

ken.johnston@aigovops.community

Shipping trust without sacrificing velocity

Sources

Otter ruling and case posture

● MLex, summary of Judge Lee’s order: mlex.com

● UC Today, what survived and what was dismissed: uctoday.com

● Mondaq, case citation 2026 WL 2351237 and claim by claim disposition: mondaq.com

● HR Executive, complaint language and the deploying employer angle: hrexecutive.com

● VoIP Review, including the Liang quote: voip.review

● CourtListener docket, In re Otter.AI Privacy Litigation, 5:25-cv-06911, N.D. Cal.

Fireflies

● Jackson Lewis Workplace Privacy Report, the three BIPA counts and damages exposure: workplaceprivacyreport.com

● Smith, Gambrell & Russell, on why deploying organizations are exposed: sgrlaw.com

● Amundsen Davis, employer facing analysis: amundsendavislaw.com

● Sheppard Mullin, practical guidance: sheppard.com

● Mason LLP, Brewer and Cruz side by side: masonllp.com