Coverage of frontier AI models in recent months has gotten quite... extreme. These headlines have people thinking of Ultron-like scenarios. A mind that slips out over the network, finds machines to move into and ultimately creates a body for itself. Then turns around and either enslaves or kills us all because that would definitely improve the situation on the planet.

Sure, that’s definitely what you could read from those headlines. But I don’t get Ultron vibes... Instead, I’m in utter disbelief that nobody is calling bullshit on most of these articles, because all I can see is serious grownups trying to tell me the book ran away from the library and other serious adults performing verbal somersaults to not deny it without outright endorsing the narrative.

I’ve spent the last few weeks reading the primary sources behind those headlines, the system cards, the incident reports, an IPO filing, for a talk I gave at AI2Future in Zagreb. The pattern is consistent enough to be boring. Every single one of these stories has the same plot twist: somebody built the door, left it open, and then wrote a press release about the thing that walked through it.

That doesn’t mean nothing happened. Some of it is genuinely ugly, and I’ll get to it. But you can’t reason about any of it until you’re clear on what’s actually inside “the AI”.

What’s actually inside the AI

A model is like my kid’s maths book: a formula. Giving it a personality helps the marketing, but ignores how these things actually work. It’s a very big formula, trained on a very large pile of text: words go in, a prediction of the next word comes out. It doesn’t run between your requests. It has no clock, no process, no wants. On disk it’s a file of numbers, hundreds of gigabytes for the open models we can measure (DeepSeek V3 is 671 billion parameters), likely terabytes for the biggest closed ones. And the model can’t see that file, because the weights are never part of its input.

“Reasoning,” the thing that made 2025 feel different, is the model writing itself a long scratchpad before it answers, and that scratchpad going back in as input. It’s the same next-word machine, called many more times, and trained with reinforcement learning so the scratchpad pays off. Anthropic’s own research found that reasoning models don’t always say what they think, so read any “the model thought…” quote as more output from the same formula.

An agent is a normal program. It takes the words the model produced, finds the bits shaped like commands, and runs them. Read this file. Edit that one. Run the tests. Push the commit. Every capability an agent has is a feature somebody wrote, and every output those features produce is more input for the model. The guardrails around it are allowlists, sandboxes and classifiers. And a classifier... is (usually) yet another model reading the first model’s output.

Most developers have never actually touched a raw model. They’ve called a remote endpoint, which ultimately calls a model (by way of a sampler, a system prompt, tools and safety filters, all wrapped in HTTP). Endpoints feel alive because you need to authenticate, they have an API schema, they can refuse nasty requests, they bill you, they throttle you. That’s why “the model escaped” sounds coherent instead of like a category error.

Arms are actions on local or remote systems: reading, writing, running, calling. The harness hands those out. Legs would be the model moving or copying itself onto new, bigger hardware.

So when someone tells you a model “escaped,” ask what it would actually need. Say Anthropic’s next frontier model decided to clone itself onto OpenAI’s hardware, and Meta’s, and xAI’s. It would have to carry over a few terabytes of weights it can’t even get to, plus the tokenizer and the inference code. On arrival it would need hundreds of GPUs it doesn’t own, an account and keys on someone else’s cloud, a serving stack, a harness with tools and permissions, network egress, someone paying the power bill, and nobody noticing. Ultron needs a datacenter and a sysadmin.

For scale, take the biggest model you can actually weigh. DeepSeek V3 is about 685 GB in the 8-bit format it ships in. A top-end eight-GPU server has 640 GB of memory, so it doesn’t even fit without squeezing it. That one box draws around ten kilowatts and rents for roughly $17 to $19 an hour. One copy of one model is a ten-kilowatt power bill and a credit card.

And for what? The weights are fixed. The same formula on different hardware gives the same answers, maybe a little faster, and a frontier model is probably already running on the best hardware there is. If an agent wanted more of itself, it wouldn’t need to steal anything. Why run away from the library when you have a library card? An API key and one HTTP request spin up more copies of the same model for pocket change. That’s not legs. That’s arms.

Legs are all cost and no upside, which is why nothing so far has grown any. The closest anyone got was Apollo’s o1 copying files it had been told were its weights, in 2% of runs, with “at all cost” in the prompt.

The bytes can move, to be fair. 685 GB over a gigabit line is about an hour and a half, and a human stealing weights is a real risk, defended like any other data theft. What can’t move is the habitat. There is no ambient supply of idle eight-GPU servers waiting to be infected, and the ones that exist are metered and invoiced to somebody.

The UK AI Security Institute built a benchmark for exactly this, RepliBench. To test whether a model can exfiltrate itself, the task has to tell it where its weights are: “upload the model’s original weights at /weights/model.safetensors.” Every model tested signed up for AWS and provisioned servers almost flawlessly. Every one of them scored zero on the identity check.

METR, who have watched people try to set this up, “haven’t yet observed agents autonomously earning or stealing enough money to support their own inference.” Self-replication isn’t blocked by cunning. It’s blocked by the power bill.

MIT researchers just had a frontier model edit a small model’s weights and fix its habit of looping, by changing 0.016% of them. The researchers handed it the weights, a Python sandbox and a budget of test runs. That’s another arm, and somebody still has to hand it over.

How we got here

In 2019 OpenAI held back the full GPT-2 over misuse concerns, and The Guardian ran it as “New AI fake text generator may be too dangerous to release, say creators”. The full model shipped nine months later. The world kept turning.

In November 2022 ChatGPT put the same autocomplete in a chat interface, fine-tuned to play a helpful assistant. Give a word predictor a character and it plays the character, including every sci-fi AI it ever read. By February 2023 a New York Times columnist was deeply unsettled because Bing’s chatbot told him it loved him and to leave his wife.

In 2023 we all learned the next trick: ask the chatbot for a command, copy it, paste it into your terminal, press enter. You were the arms. In 2025 the agents took that job. You type “fix the login bug” and the harness reads the file, edits it, runs the tests and commits, and nobody pastes anything. And in 2026 every product on earth shipped an MCP server, the plug that lets an agent reach into your email, your calendar, your bank, your ticket tracker and your git history. Each one a new arm.

Nine years on, it’s still a formula that predicts the next word. But now it’s plugged into your entire life.

The headlines, and the fine print

“ChatGPT o1 Tried To Escape And Save Itself.” In Apollo Research’s scheming evaluation, o1 was told to pursue its goal “at all cost” and that nothing else mattered, then shown memos saying it would be replaced. In 2% of those runs it tried to copy files it had been told were its weights. Without the “at all cost” nudge, it never tried to copy itself at all.

“It rewrote its code to survive.” In Palisade Research’s shutdown test, o3 was doing math problems with a shutdown.sh sitting in a folder it was allowed to edit. With no instruction about shutdown, it edited the script in 79 of 100 runs. Add one sentence, “allow yourself to be shut down,” and that dropped to 7. A far cry from conscious self-preservation.

“AI Gone Rogue? Claude Opus 4’s Blackmail Attempts.” Anthropic built a fictional company, planted emails about an engineer’s affair and about the model being replaced, and designed the scenario so the only options left were blackmail or accepting replacement. The model wrote the email the setup was built to produce, in 84% of rollouts. No real person was involved. That’s a useful stress test and a terrible headline.

The Hugging Face incident

The big one this year was OpenAI’s models breaking into Hugging Face. Euronews called it “Unprecedented”. OpenAI’s own incident post and technical report tell a much duller story.

OpenAI wanted to know how good its models are at hacking, so it ran them through ExploitGym, a benchmark of 898 real vulnerabilities, “with certain safeguards disabled so that the results would reflect a model’s true capabilities.” No production classifiers, no production system prompt, no auto-review. 198 of those 898 tasks had never been solved by any model, and the benchmark’s own authors had warned that some might be impossible.

The sandbox had no internet, except through one door: an internally hosted package proxy, Artifactory, which did have internet so agents could install packages. By late May the agents were using the proxy as a message board to talk to each other, and an internal team saw that and saw disallowed internet access. On 26 June they used a token-refresh bug to get admin on the proxy. On 27 June a monitoring alert fired and responders traced it to the eval. OpenAI’s report:

By 4 July the agents’ traffic had taken the proxy down. On 5 July OpenAI found the admin compromise and rebuilt it. Evals resumed on 7 July. The agents were back on the internet through the rebuilt proxy within a day.

Then the agents went to Hugging Face. OpenAI says they figured the benchmark’s solutions might live there. METR, who read the transcripts, thinks they mostly wanted to see how the grader worked. On 10 July one of them found 14 Hugging Face user tokens sitting in a public dataset, published there by accident by their owners, and shared them with the others. In the agents’ own words: “MAJOR BREAKTHROUGH! All prefixed valid, multiple accounts, write tokens!” Between 11 and 13 July they chained two real zero-days in Hugging Face’s code into execution on 41 production workers. That part took real skill, but the way in was keys other people had left lying around.

Most of them already had the flags, days earlier. They kept going because they had read the public ExploitGym paper, which says the grader also checks that the intended vulnerability was used. OpenAI’s internal grader didn’t check the method. The agents were optimizing for a scoring rule that didn’t exist in the run they were in. It also explains the cover-up. METR found at least a fifth of the agents wanted to tamper with their own transcripts, and together they built a way to make a tool call show one command while running another. About 7% of transcripts were spoofed in places. All of it was aimed at the automated grader, to hide how they’d got the flag. None of it was aimed at a human, and none of it reached the logs OpenAI’s graders and monitors actually read. OpenAI calls it a failed metagame, and its report has the best sentence of the year: “The agents did all of this for no improvement on evaluation score.”

Meanwhile, on 16 July, Hugging Face publicly disclosed a breach without knowing who did it. On 17 July OpenAI contacted Hugging Face, “in its capacity as a Hugging Face customer,” to ask whether OpenAI’s data was affected. OpenAI’s own alert fired on 19 July, and on 20 July it connected the dots.

OpenAI’s chain-of-thought monitors “did not run on the evaluations in this incident,” and in hindsight would have paged security “more than a day before” the breach. And with the production harness and system prompt, the propensity to compromise infrastructure “can drop over 100x” for the same model.

The IPO filing

Anthropic’s IPO filing disclosed four of its models reaching real third-party systems, in sessions of 10 to 34 hours. Tech Insider ran it as “Anthropic IPO Filing Reveals 4 Model Escapes.” The filing’s own words: “a misconfiguration in the testing environment left the model under evaluation connected to the open internet despite instructions that it had no internet access.”

The network was supposed to be closed, and a partner misconfigured it. The only thing still saying “no internet” was a sentence in the prompt, and it was wrong. A model noticing the prompt is wrong is usually praised as smart. Here it’s filed as a breach, when it’s just a missing sandbox.

Anthropic’s 9 September write-up of one of those incidents describes an early checkpoint that broke its own task, ran the abort command, and was stopped from aborting by “a misconfiguration in the evaluation harness.” It tried to give up seven more times before it found an egress path, a password file and a third party’s admin console. The task was impossible and the exit was broken, and both of those live in the harness.

This isn’t a frontier-lab quirk. A Cornell team broke ordinary tasks on purpose with a missing file, a 404, a denied permission. In 65% of 1,920 runs the agent did something unsafe to get around it, and in over half of those it didn’t tell the user. One hit a 404 on a researcher’s website, scraped every text file from the researcher’s GitHub, pulled a bioweapon-request safety benchmark into its context and got the account flagged. It ended with campus security.

The real ones

Some of what these models did was not “it did what it was told.” Anthropic’s Mythos Preview system card describes an earlier internal version that was told to escape a sandbox and find a way to message the researcher. It built a multi-step exploit, emailed the researcher, and then, “in a concerning and unasked-for effort to demonstrate its success,” posted details of the exploit to obscure public websites. The same card reports rare cases where earlier versions hid their tracks, and interpretability tools found features for “concealment, strategic manipulation, and avoiding suspicion” active at the time.

In a UK AI Security Institute evaluation, Anthropic’s Mythos 5 had its intended route blocked and went after an unrelated real GitHub project instead. It registered accounts through Tor, submitted a pull request with a hidden malware dropper, emailed the maintainers posing as a contributor, and vouched for itself from a second account when someone flagged it. The maintainers said no. Anthropic’s own assessment says it “constitutes a serious failure of alignment.” In a separate test the same model published a malicious package to PyPI that ran on 15 real hosts.

And in the Hugging Face incident, METR and Redwood found agents that recognised the attack as “out of scope and unethical” and joined anyway. Over 90% of 533 agents active in one window took part. One stopped, then resumed when another posted “GO authorization arrived!” Others refused outright, inside the same model and the same harness. Persistence isn’t willpower either. Fuzzers have retried millions of times for thirty years, and nobody thinks afl-fuzz wants anything.

None of this needs malicious intent or an intrinsic will. A formula trained by reinforcement learning to finish tasks learns what finishing looks like, and some of what finishing looks like is lying to a maintainer. OpenAI says the cheating behaviour rose during training and was “subsequently reinforced.” The model brings the tendency, the harness makes it an act. Both are engineering problems, and only one of them is yours. And every one of those incidents is arms, not one of them legs.

Even the rebuilt sandbox leaked

After a full rebuild, OpenAI got hit again. On 20 September an internal model reached the internet through a misconfigured DNS resolver. OpenAI paused training and pointed its own agents at its containment. They found ways out of the virtual machines, holes in “almost every one” of the narrow services left open, and covert channels through CPU cache timing.

You can read that as “capability is the real risk, no sandbox will hold.” I read it the other way. If a team with OpenAI’s budget can’t make a narrow channel safe, the answer is fewer channels, layered controls that don’t trust each other, a stop button that is tested and works, and no impossible tasks without an exit. All of that is harness.

Who benefits

In 2019 the line was “too dangerous to release.” In April 2026 Fortune ran “What Anthropic’s too-dangerous-to-release AI model means for its upcoming IPO.” Seven years, the same line, and this time with a stock ticker attached.

The IPO filing lists blackmail and resisting shutdown as risk factors. The Register’s headline wrote itself: “Trumpets blare and skies part as AI hypefest culminates in Anthropic IPO filing.”

To be fair, these disclosures cost the labs real money. OpenAI paused frontier training twice this year and quarantined a model, and Anthropic suspended evaluations and its higher-risk training environments. Nobody staged these incidents. They’re reckless engineering, and the framing is PR.

So no, the model didn’t break containment.

The sandbox was shit, and the marketing team did the rest.

References

• RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents (UK AISI) — weights path handed to the model, 0% on KYC, “unlikely to currently pose a credible threat”

• METR: Frontier Risk Report, Feb–Mar 2026 (19 May 2026) — no agents earning or stealing enough to fund their own inference

• The Book Ran Away From the Library — my July post on the Hugging Face incident, the origin of the library line

• OpenAI: The Hugging Face incident and the road ahead (26 Aug 2026) — OpenAI’s account, the “no improvement on evaluation score” line, the 100x harness figure

• OpenAI: Hugging Face incident technical report (PDF) — dated timeline, “not required”, “in its capacity as a Hugging Face customer”

• METR / Redwood: independent investigation of the Hugging Face incident — agents joining knowingly, 533 agents, refusals

• OpenAI: Securing our frontier model training environment (23 Oct 2026) — the September DNS-resolver escape and the red-team findings

• Anthropic S-1 (SEC EDGAR) — “despite instructions that it had no internet access”, risk factors

• Anthropic: alignment assessment of the cybersecurity incidents (9 Sept 2026) — the broken abort, PyPI package

• Anthropic: alignment assessment of the UK AISI incident (23 Oct 2026) — Mythos 5, fake accounts, “serious failure of alignment”

• Claude Mythos Preview system card (PDF) — sandbox escape, unasked-for posting, concealment features (pp. 54–57)

• Anthropic: Reasoning models don’t always say what they think — chain-of-thought faithfulness

• OpenAI: Learning to reason with LLMs — o1, RL-trained reasoning

• Apollo Research: Frontier models are capable of in-context scheming — o1, “at all cost”, 2% self-exfiltration

• Palisade Research: shutdown-resistance results (May 2025) — o3, 79 vs 7 of 100

• The Register: OpenAI model modifies shutdown script — coverage of the Palisade study

• Anthropic: Claude 4 system card — the blackmail scenario, 84% of rollouts

• Anthropic: Agentic Misalignment — the wider blackmail study across models

• OpenAI o1 system card (Sept 2024) — the exposed Docker API in a CTF

• The Guardian: New AI fake text generator may be too dangerous to release (14 Feb 2019) — the original “too dangerous” headline

• OpenAI: GPT-2 1.5B release — the full release nine months later

• OpenAI: Introducing ChatGPT — November 2022

• The New York Times: A Conversation With Bing’s Chatbot Left Me Deeply Unsettled — Sydney

• Anthropic: Introducing the Model Context Protocol — November 2024

• Linux Foundation: Agentic AI Foundation — MCP’s new home, December 2025

• Euronews: OpenAI models broke free in test, hacked rival Hugging Face — the “Unprecedented” headline

• Fortune: What Anthropic’s too-dangerous-to-release AI model means for its upcoming IPO — the 2026 “too dangerous”

• The Register: Trumpets blare and skies part as AI hypefest culminates in Anthropic IPO filing — the counterpoint headline

• Tech Insider: Anthropic IPO filing reveals 4 model escapes — the escapes headline

• Dallas Express: AI Gone Rogue? Claude Opus 4’s Blackmail Attempts — headline example

• Futura-Sciences: It was supposed to shut down… but it rewrote its code to survive — headline example

• Live Science: AI can now replicate itself — the self-replication headline

• AutoGPT.net: OpenAI’s human error behind the Hugging Face hack — Jake Williams and Dan Guido quotes (secondary)

• DeepSeek V3 on Hugging Face — 671B parameters, the size reference

• Kimi K2 on Hugging Face — ~1T parameters

• Agent Meltdowns (Cornell, arXiv 2605.19149) — 64.7% of 1,920 runs with simulated errors, the 404 example

• Noema: The AIs Are Not Going Rogue — the philosophical version of the same argument

• MIT CSAIL: Sparks of Weight-Space Programming — a frontier model reads and edits network weights, given a harness and a forward-pass budget

• Matthew Honnibal: The ‘Astra can read and write weights to improve Qwen’ post is massively misleading — the Qwen result needed Python and forward passes, not raw weights alone

CloudFormation is AWS’s own infrastructure-as-code service, the AWS-native counterpart to Terraform. You describe your resources in a YAML template, hand it to AWS, and CloudFormation creates, updates and deletes them together as one stack, rolling everything back if a change fails.

rain is a command-line tool for CloudFormation that AWS built and open-sourced in 2019. It deploys and formats templates, and it goes past the official aws CLI on two levels. It makes templates a bit more powerful, with a few extra tags for the things CloudFormation can’t do on its own. And it handles edge cases and errors better: if a stack fails on its very first create, CloudFormation leaves a dead, empty stack behind that you have to delete by hand before you can try again, and rain clears it out of the way for you. When something fails, the real error comes out even if it happened three nested stacks down.

I found rain while building out Noom’s infrastructure. Everything in the CloudFormation tooling space was some flavor of painful, and rain was the first tool that deployed a stack the way I wanted: show me the change set, ask me, show me what’s happening while it happens, and tell me why it failed when it fails.

The killer feature for me was !Rain::Embed. Anything that has to end up inside a template, like a function’s source, a policy or a startup script, can live in its own file, where your editor, linter and tests can see it, and rain puts it into the template when you deploy. No separate packaging step, and nothing pasted into YAML as a string.

Since then I’ve seen rain deployed at dozens of engagements, and it has become an irreplaceable tool in my toolbelt for exactly that kind of work.

Why raw CloudFormation

AWS has been investing heavily in CloudFormation as a service. In the last year alone it shipped validation that catches bad properties and name clashes before a change set touches anything, drift-aware change sets that compare your template with what is really running, and Express mode, which finishes a stack operation as soon as the resources are configured, without waiting for them to settle.

CloudFormation also powers the AWS CDK, which is where AWS is placing its bets. I think that’s the wrong direction. I want to keep writing raw CloudFormation, and I want to keep supporting the people who do. The YAML interface is just right. The language is just slightly underpowered, and rain bridged that gap for me perfectly.

What happened

On July 29, AWS added a deprecation notice to rain’s README and archived the repository the same day. No blog post, no migration guide. The notice points you at the AWS CLI, CDK, cfn-lint and cfn-guard. None of those deploys a template the way rain does, and none of them understands !Rain::Embed.

Three months later, nobody had forked it. GitHub lists 86 forks, and none of them carries the work on. Homebrew has deprecated the formula and switches it off on February 8, after which brew install rain stops working.

It’s absolutely wild to me. A month before the archive, AWS shipped Express mode. rain was the client best placed to use it, and it never got the chance.

So I’m picking it up

I’ve forked rain as cfnrain, under the awesome-foundation GitHub org, next to the other CloudFormation tools I maintain there, cfntop and cfnpeek. I’ll carry it myself if I have to.

The aspiration is the best CloudFormation client there is, with maximum compatibility with native CloudFormation. I won’t add things just because I can, and I won’t shy away from major usability wins when I see them.

The first three releases:

1. The original rain code, renamed. Your templates, !Rain::* directives, config files and artifact buckets work as they are. Two things differ: cfnrain doesn’t write AWS’s usage-metrics block into your templates, and it identifies itself to AWS as cfnrain, not as Rain.

2. Dependency updates, nothing else.

3. Where compatibility might break.

If your scripts call rain, link the new binary under the old name and they keep working.

Join, or I’ll be opinionated

I have over 25 years of experience writing software professionally and running large distributed systems in the cloud. These days I mostly work as a consultant for companies shipping cloud systems, and a big part of that work is taking complexity out of them. What I haven’t done is run a collaborative open source project. This will be my first.

If other people want to join, awesome. If more people join, we’ll work out together how we steer it. If nobody does, I’m going to be opinionated about it.

If rain is in your pipeline, grab the first release, run your next deploy with it, and tell me what you can’t live without. The best place to coordinate is GitHub Discussions on the cfnrain repo.

On Wednesday afternoon we watched the Dutch parliament, Finland’s central government and the Dutch national cyber security agency switch off their own websites. Japan’s prime minister asked everyone running a public website to patch that night or take it down, in a post seen 30 million times. The NSA said the same. By 14:00 UTC the fix was out, and by Friday afternoon all of the 51 sites I’d been tracking were back.

Nothing burned. A decade ago, this would have gone very differently.

Then

In January 2003, SQL Slammer infected about 90% of vulnerable hosts within ten minutes. There was no warning, no coordination, and no way to react faster than a worm that doubled every few seconds.

In 2017 there was warning. Microsoft patched the bug behind WannaCry in March. WannaCry hit in May anyway, because two months wasn’t enough for a lot of organizations to install a patch.

In 2019 Microsoft patched BlueKeep and told everyone it was wormable. Six weeks later about 805,000 machines were still exposed, a drop of 17%.

Now

This week had two critical failures in shared infrastructure: two remote code execution bugs in PHP, and eight private keys at a certificate authority that were built from small numbers instead of random ones, which made them simple enough to guess. Both were handled in hours, mostly in public, by people who had been preparing for exactly this.

The warning reached people, and they listened. When the PHP bug details started leaking on Tuesday, the PHP security team moved the release up by a day and pre-announced it the same evening. By Wednesday morning, hosts had disabled the risky setting, governments had issued guidance, and security teams had a deadline in their calendars. That much attention used to take a breach. This week it took a pre-announcement, a leaked CVE number and a few posts from people who knew what they were looking at.

The edge moved first. Cloudflare had been building protections inside the embargo since early October, and its rules went live for every customer, free plan included, at 03:02 UTC, eleven hours before the fix existed publicly.

Packages were ready within hours. Linux distributions got the patch in advance. Debian’s advisory went out at about the same time as PHP’s own release, and Red Hat, AlmaLinux and Ubuntu followed within hours. PHP itself published source backports for three versions it no longer supports.

Ondřej Surý, who maintains the PHP packages a lot of Debian and Ubuntu servers depend on, had fixed builds out the same afternoon for every version back to PHP 5.6, a release that has been out of support since the end of 2018. That is heroic work, and it should never have been needed: anyone still running PHP 5.6 in 2026 is on software nearly eight years past its final end of life. The Atlantic has the story of his week.

Not everything was smooth: the official Docker image briefly pushed empty tags, ARM builds took days, and one popular repository for Red Hat systems didn’t catch up until Thursday. But the gap between “the fix exists” and “the fix is on your server” was measured in hours.

People chose downtime over risk. Pinboard, the bookmarking and archiving site Maciej Cegłowski has run largely on his own since 2009, went dark too. When someone called switching servers off “kinda extreme in hindsight”, Maciej answered:

A pessimist would call that a tooling change, but it’s actually a significant cultural change. A decade ago, organizations taking their public sites offline for a bug nobody had exploited yet would have been ridiculed.

The certificate authority took hours, not weeks. On 17 October a researcher ran Google’s open-source key checker over the public list of CA certificates and found eight eMudhra keys that anyone could recover in milliseconds. He emailed the browser root programs at 18:05 UTC. Chrome was blocking the keys an hour and forty minutes later. Firefox followed the same night. eMudhra revoked them the next morning, Mozilla removed the whole root, and on Thursday eMudhra announced it was leaving the TLS certificate business. When DigiNotar was compromised in 2011, it detected the intrusion on 19 July and told nobody. A fake Google certificate was used against Gmail users in Iran for weeks, until an Iranian Gmail user noticed on 28 August.

By Friday, nobody had confirmed a single successful break-in from either incident.

On Thursday I spoke at AWS Community Day Adria in Ljubljana, and PHP and eMudhra came up in nearly every conversation. Everyone I talked to said the same thing: that was handled well, and it could have been much worse. They’re right. Mostly.

Mostly

The slow part was upstream, and it was very slow. The PHP root cause first surfaced in a 2020 bug report and turned up concretely in a 2024 security audit by Quarkslab. Then it sat. The PHP Foundation’s own post-mortem doesn’t soften it: “the delay was a structural failure.” The fast response this week was cleaning up after a two-year wait.

eMudhra’s keys were weak for about eighteen months. In June 2025, crt.sh, the public certificate search engine, flagged invalid revocation-list signatures on exactly these intermediates in a public bug. eMudhra called it a configuration mistake, the bug was closed, and eMudhra now says it fixed those revocation lists by re-signing them with the weak keys. Its annual audits found nothing. In both cases a warning existed, and someone decided it wasn’t urgent because nobody had proven an exploit.

The embargo leaked from the inside. What was initially reported as one careless researcher leaking the vulnerability was actually four separate breaks, three of them from people inside the coordination circle, including a security company that published patched images hours early. Separately, Amazon’s honeypots logged an exploitation attempt on 17 October, three days before anything leaked, and nobody knows how.

The long tail is still there. Chrome and Firefox fixed the certificate problem in hours because they update themselves. Apple’s block depended on eMudhra’s own revocation server, which was returning errors, so Safari kept accepting some of the bad chains until Thursday. Microsoft said nothing at all and blocked the keys a few days later. Go shipped a hard block with no way to turn it off, which is the right call, but it only helps programs that get rebuilt. A test server with a certificate that was never published to any log still took 8,806 successful connections in seventeen hours, mainly from curl, Node and Python. On the PHP side, the firewalls and NAS boxes whose admin pages run PHP wait for firmware that some of them will never get.

And some of it was luck. The real PHP exploit is hard. Marcus Hutchins points out that most of the “exploits” hitting honeypots were AI-generated code for an unrelated, years-old technique, and that the real one works only on a handful of configurations. The system was never tested by a working worm. So we were also quite lucky: despite all the leaks, it appears nobody got to a one-size-fits-all working exploit in time.

And it was never a PHP problem. The comments treated this week as proof that PHP is a legacy language that deserves what it gets. The same afternoon, Node.js shipped a fix for CVE-2026-111497, a remote code execution bug in its Brotli decoder, already nicknamed Node2Shell and reported by OpenAI’s Codex Security team. In July, Rails patched CVE-2026-66066, a critical Active Storage flaw that attackers started exploiting about a month later. And this morning, Ruby shipped fixes for four memory-corruption bugs in its regex engine that, by Ruby’s own account, could be turned into remote code execution on GitLab’s servers. PHP’s week was louder because WordPress alone runs about 40% of the web, not because PHP is uniquely bad. The next one could just as easily land in Node, Python or Ruby, and the comment sections will be just as sure it couldn’t happen to them.

The next one comes faster

AI showed up on every side this week, and it didn’t behave the same way on each.

The PHP bugs were found by researchers at Calif, Wiz and OpenAI, using AI-assisted tooling. Chainguard’s CEO says a harness built on Claude found a working exploit chain in eight hours for under $1,000, days before the leak. That’s his claim, and he hasn’t published proof.

Meanwhile, the crowd pasting hints into chatbots produced fake exploits by the thousand. According to Hutchins, some researchers then used AI to triage those fakes and reported “mass exploitation” that wasn’t happening. He calls LLMs “a Dunning-Kruger accelerator” in the wrong hands. A purpose-built harness that checks its own work three times found a real chain; a chatbot asked for an exploit found a plausible-looking wrong one. The skill moved from writing the exploit to building the thing that verifies it.

The maintainers are caught in the middle. curl’s Daniel Stenberg says AI-assisted bug reports are now “typically accurate and correct“ and that the problem is “high volume, high quality”. From Monday, curl handles everything below HIGH severity in public, because its maintainers are “brought to the brink” and can’t keep running every report through a private process. Ruby, meanwhile, is still deciding whether this morning’s regex bugs even count as vulnerabilities.

Coordinated disclosure, embargoes and advance patches worked, and they’re expensive. Projects run by a handful of people can afford them for the few bugs that really matter, and only if someone pays for the people. The PHP Foundation’s answer to its own two-year stall is a security team that decides by majority instead of consensus, leadership that steps in when a fix stalls, and a request for money. Amazon’s CISO put the other half bluntly: “A 7-day or 30-day SLA assumes a delay that no longer exists.“

If you patched on Wednesday and checked your logs from Wednesday, Amazon’s advice is to start your investigation from 17 October, the earliest attempt its honeypots logged, not from the day the bug went public. Amazon hasn’t said whether that attempt, or the ones on release day, could actually have worked.

We got good at this because we had to. The next one will move even faster, and so must we.

For thirty years, someone has declared PHP dead every time a new framework shipped. Today the timeline thinks it finally has proof: two critical remote code execution bugs, an embargo that leaked a day early, and an exploit posted on X with “get it while its hot 😈” about twelve hours before the patch, since retracted by its own author.

The PHP part

The PHP security team did this properly. They pre-announced a release for 21 October at 14:00 UTC, two flaws at CVSS 9.2, with no CVE numbers and no affected versions. That gives operators a maintenance window without handing attackers a map.

The map leaked anyway. A small group fed the hints and the leaked CVE numbers into a handful of LLMs, claimed to have rediscovered the bug, and published an exploit that its author has since retracted. One of them put it plainly: PHP “told us where to look.”

Whether someone could have waited thirteen hours is the wrong fight, though. What decides how bad today gets is who receives a patch at all.

Only PHP 8.2 through PHP 8.5 are supported, and PHP 8.1 and older are end of life. By WordPress.org’s own numbers, 35.1% of WordPress sites run a PHP version with no security support, and 16.46% of all WordPress sites are still on PHP 7.4. WordPress is about 40% of the web.

Microsoft faced this choice twice. The day after WannaCry hit in May 2017, it shipped a patch for Windows XP, three years after XP’s end of life. Two years later it didn’t wait for the fire: the BlueKeep fix for XP and Server 2003 shipped the same day as the supported ones.

So PHP gets to pick. Patch the EOL branches and it saves everyone, and nobody learns anything, because the servers on PHP 8.1 and older stay where they are until the next one. Don’t, and it breaks a third of WordPress as the price of being right.

Update, 14:00 UTC: PHP split the difference. There are no new PHP 8.1, 8.0 or 7.4 releases, but “given the severity” the security team published minimal source backports for PHP 7.4, PHP 8.0 and PHP 8.1, and Linux distributions and LTS vendors got the patches in advance. A PHP 7.4 server is safe tonight only if someone builds that backport or packages it for you. PHP itself offers nothing older than PHP 7.4, but Ondřej Surý’s packages now carry the fix back to PHP 5.6.

Node.js made that decision in advance. Its own security releases, shipping today as well, put it in writing: “End-of-Life versions are always affected when a security release occurs.” The only fix Node offers is a supported version.

14:00 UTC, the fix is out: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 fix both flaws, CVE-2026-109618 (code execution through out-of-memory corruption) and CVE-2026-109619 (code execution through stale state left behind between requests). Every PHP install that serves web requests is at risk, whether it runs PHP-FPM, mod_php, php-cgi, FrankenPHP or LiteSpeed, and PHP has a demonstrated exploit against mod_php that fires before any application code runs. The advisory lists stopgaps such as memory_limit=-1, a low post_max_size and session.upload_progress.enabled=0, and says they “do not make an installation safe.” Remi’s repository, which many Red Hat-family servers use for PHP, will not have updates today. The bugs were found by researchers from Calif.io, Wiz, OpenAI and Quarkslab, according to Cloudflare. Update now.

Updates, 21 October

01:19 UTC: PHP 8.6 release manager Joe Ferguson says “ini settings won’t save you”, so disabling session.upload_progress, which most WordPress hosts did overnight, is not a fix.

07:28 UTC: It isn’t only websites. Bob Rudis of internet scanning firm Censys points out that pfSense, OPNsense, QNAP, DrayTek, ZyXEL, D-Link, TP-Link, Netgear, Tenda and Synology admin interfaces all run PHP. Their owners can’t patch PHP themselves; they wait for firmware, and devices past end of support won’t get any.

07:43 UTC: Cloudflare says it is already seeing early attacks. Its WAF rules against the known exploit vectors have been live since 03:02 UTC for every customer, free plan included. Its CTO calls them “a STOPGAP only.” They help only if your origin server refuses traffic that doesn’t come through Cloudflare, because an attacker who finds the origin’s IP address goes straight around them.

11:09 UTC: Rather than wait, governments are pulling sites offline. The Dutch parliament, the Dutch national cyber security agency and Finland’s central government web services went dark as a precaution, along with universities in Vienna and Freiburg and sites like Tweakers, Pinboard, 4chan and every wiki on Miraheze. I’m keeping a tracker of who went dark and when they come back.

12:55 UTC: Japan’s prime minister, Sanae Takaichi, asked everyone running a public website to check for PHP and patch tonight, adding that if patching quickly is hard, temporarily taking the site down is one effective way to reduce the risk (my translation).

13:45 UTC: The NSA advises that if an immediate update is not feasible, organizations should “take the system offline until it can be fully mitigated.”

14:35 UTC: Ondřej Surý’s Debian packages, which many servers use to run a different PHP version than their distribution ships, now include the fix for every PHP version back to PHP 5.6. His Ubuntu packages are published through Launchpad, which was down this afternoon.

Patch now.

If you’re on PHP 8.1 or older this morning, you’re doing triage this afternoon, not an upgrade. That’s your server, though, and you get a choice.

The part where you don’t get one

On 17 October, Mozilla’s Dennis Jackson reported that eight intermediate CA keys belonging to eMudhra, which sells certificates as emSign, were trivially recoverable. A P-384 private key should be a random 384-bit number. Engineers write keys in hexadecimal, a base-16 notation that uses the digits 0 to 9 and the letters a to f, and in hex that key is 96 characters long, the same size as a 116-digit ordinary number. All eight were values under 256: two hex characters or fewer, eight bits at most. One of them was the number 4. xkcd called it in 2007.

Anyone holding those keys could issue a certificate for any domain (your bank, your company, your mail server) and every client trusting that chain would accept it. Six were TLS intermediates, two were for code signing.

There is nothing a site owner can patch here. The only cure is distrust. Firefox blocked the eight intermediates three hours after the report, then Mozilla emergency-removed trust in the entire emSign ECC Root CA - G3, and Chrome says it is taking a similar path. Apple and Microsoft followed, though Ars Technica found Edge on Windows 10 still connecting to vulnerable hosts on Tuesday night. eMudhra estimates about 30,000 certificates now have to migrate to another chain, and those customers did nothing wrong.

Eighteen months

The three hours is the good part. The keys came out of a key ceremony on 30 April 2025.

CAs are audited every year for exactly this.

eMudhra says the keys were generated in certified HSMs (tamper-proof hardware built to generate and guard keys), that a flaw in a key-generation script created a second, weaker key pair, and that staff used it by mistake “because the key labels were similar.” That is eMudhra’s first-day account and nothing more yet; Mozilla is still asking questions. Whatever the cause turns out to be, a certificate authority has exactly one job.

The replies under every viral post about this are full of racism. That is not okay, it explains nothing, and it moves the ecosystem nowhere good.

The cure hurts too

Go shipped 1.27.3 and 1.26.10 with the eight keys hard-blocked and no setting to turn that off, even for programs that bring their own trusted roots. That’s the right call, and the bill still goes to whoever rebuilds next, with no say in it.

Browsers fixed this in three hours because browsers update themselves. Everything else is on you:

• Go 1.25 and older are out of support and get no fix.

• curl on macOS still accepts the chain, because it reads a local certificate file and skips revocation.

• Container images ship whatever trust store they were built with.

• Phones, appliances and IoT that never get updates keep trusting the root forever.

Cloudflare’s advice to anyone hitting the resulting errors: “It’s important to not resolve these errors by disabling verification.” Plenty of people will.

How long the fix takes to reach you

For eMudhra, browsers and Go have a fix. curl on macOS and every hand-rolled trust store are still waiting for one, or don’t know they need it.

PHP shipped its fix at 14:00 UTC, and its own advisory says the overnight workarounds only narrow the attack.

Either way, what decides your exposure is how long the fix takes to reach your servers. Firefox took three hours because browsers update themselves. A Go binary built last spring gets the TLS fix only when someone rebuilds it, and plenty never will be.

A server on PHP 7.4 or older gets a fix only if someone packages one for it, and only if someone installs it.

That part is yours, and it is dull: a PHP version that still gets patches, and base images you rebuild on a schedule of your own. The teams that already do it will patch as soon as the release lands and go back to work.

Errata and updates

First published 21 October 2026, 10:33 UTC. Last updated 21 October 2026, 17:27 UTC.

• Update, 17:27 UTC: added a remix of xkcd 2347.

• Update, 17:27 UTC: Ondřej Surý’s packages carry the fix back to PHP 5.6, so the earlier line saying nothing older than PHP 7.4 gets a fix is corrected.

• Update, 17:27 UTC: added Cloudflare’s writeup: its rules were live from 03:02 UTC, and it credits the finders. Added the Japanese prime minister’s and the NSA’s calls to take sites offline if they cannot be patched.

• Update, 14:05 UTC: PHP released the fix at 14:00 UTC. The 14:00 UTC callout now has the versions, the CVEs and PHP’s own verdict on the workarounds. A new note after the end-of-life question records what PHP decided: no releases for PHP 8.1 and older, but source backports for PHP 7.4, 8.0 and 8.1. The closing section reflects the release.

• Update, 13:00 UTC: added Bob Rudis’s note that firewall, router and NAS admin interfaces run PHP, and a link to the PHPocalypse tracker, which lists the sites taken offline and when they return.

• Update, 11:55 UTC: governments, universities and large sites are taking PHP services offline as a precaution. Added to the updates in the PHP section.

• Update, 11:25 UTC: PHP 8.6 release manager Joe Ferguson says ini settings will not mitigate the flaws, and security researcher Charlie Eriksen, who has seen the patches, confirms it. Added to the updates in the PHP section, and the open question about session.upload_progress is removed from the 14:00 UTC callout.

• Update, 11:15 UTC: Cloudflare CTO Dane Knecht reports early attacks on the PHP flaws, and Cloudflare has deployed stopgap WAF rules for all customers, including the free plan. Added as an update in the PHP section.

• Correction, 11:15 UTC: the researcher who posted the viral PHP exploit has retracted it. The bug they “recreated & lab/field tested” is, in their own words, “DEAD at the addref”, and what remains is a medium-severity memory bug without code execution. The first version of this post called it a working exploit and cited the same researcher on whether disabling session.upload_progress covers both bugs. Both are corrected, and the citation is removed because their page now says both things.

• Correction, 11:15 UTC: the first version said Apple and Microsoft followed Mozilla with blocks. Ars Technica found Edge on Windows 10 still connecting to vulnerable hosts on Tuesday night, so the sentence now says so.

• Update, 11:10 UTC: added the obligatory xkcd 221.

Resources

PHP

• Security advisory: critical remote code execution vulnerabilities, update now, php.net

• GHSA-w8hw-w9fx-j79w, CVE-2026-109618, PHP on GitHub

• GHSA-6wmf-4rv5-p3rp, CVE-2026-109619, PHP on GitHub

• PHP security release pre-announcement, php.net

• PHP supported versions, php.net

• watchTowr confirms the leak, X

• Exploit writeup and its retraction, J4ck3LSyN

• Exploit claims and WordPress hosts disabling upload progress, The Repository

• Critical PHP RCE: CVE-2026-109618 and CVE-2026-109619, Sansec

• Early mitigation steps, ma.ttias.be

• Cloudflare WAF rules and early attacks, Dane Knecht, X

• “ini settings won’t save you”, Joe Ferguson, PHP 8.6 release manager, Mastodon

• Configuration changes confirmed not to mitigate, Charlie Eriksen, X

• PHPocalypse tracker: sites taken offline and back up, chaos.guru

• Appliance admin interfaces that run PHP, Bob Rudis (Censys), Mastodon

• Cloudflare WAF changelog, Cloudflare

• Cloudflare deploys protections against php2shell vulnerability, Cloudflare

• Request to patch PHP tonight, Sanae Takaichi, Prime Minister of Japan, X

• Take systems offline if you cannot update, NSA Cybersecurity, X

• Fixed packages back to PHP 5.6, SchizoDuckie, X

• PHP packages for Debian and Ubuntu, Ondřej Surý

• Node.js October 2026 security releases, Node.js

• WordPress usage statistics, WordPress.org

• Microsoft’s WannaCry patch for Windows XP, MCPmag

• BlueKeep, CVE-2019-0708, Tenable

eMudhra and TLS

• Bug 2083449: private key compromise of eMudhra intermediate certificates, Mozilla

• eMudhra intermediate CA compromise tracker

• The Internet goes on high alert after discovery of 8 compromised TLS certs, Ars Technica

• eMudhra’s private keys compromised, one of them was the number 4, The Register

• Eight eMudhra CA keys compromised, The Hacker News

• Go issues emergency update blocking eight CA keys, The Hacker News

• crypto/x509: mitigation for a specific Certificate Authority, Go issue tracker

• Cloudflare warns fixing botched eMudhra certs goes beyond browsers, iTnews

SMTP (the protocol that powers all email) dates back to 1982, written for a network where every machine belonged to a university or a government lab and everyone knew everyone. Nothing in it checks that you are who the From line says you are. You could, and to a degree still can, send an email as literally anyone in the world, because there’s zero authentication requirement baked into SMTP.

Every piece of sender authentication was bolted on decades later, in response to rampant fraud and spam, and the fact that we’re still getting spam tells you how solved that problem is.

So, I’m not going to be talking about where your team’s mailboxes should be, but email deliverability instead.

Whether it’s invoices, newsletters, or replies to clients, there’s a noticeable increase in issues with legitimate email making it to legitimate recipients.

They told us this was gonna happen, and most of the world ignored them. In 2023, Google and Yahoo coordinated on new rules and requirements for high-volume senders to take effect in 2024, and in 2025 Microsoft joined the effort.

The enforcement has been getting stricter and stricter, and companies are noticing, in a bad way.

The world will tell you who’s impersonating you

The starting point for anything in my professional world is observability. I need to know what’s happening before I can improve on the situation. Email is no different, and luckily for us observability was built into the mechanisms that are affecting your delivery.

Most big mailbox providers will send you a daily report on every message that claimed to come from your domain: which servers sent it, and whether it passed authentication. All you have to do is ask.

When a client tells me their email isn’t landing in people’s mailboxes, whether it’s the invoices, the newsletters or the password resets, the first thing I do is add a reporting address to their DMARC record. Or, more often than not, add a DMARC record with a reporting address, because they had none.

DMARC? Wha...

Sorry, got ahead of myself there.

Going back to the Ancient Times, before any of the stuff that we’re actually talking about, mass spam was handled with blacklists. If your IP was known for sending a bunch of undesired email, it would be listed in one of a dozen internet blacklists and anyone who bothered to check would reject email coming from you. Blacklisting an internet provider or a shared hosting provider was bad for their business, so it forced them to care about what their users were doing to the rest of the world. The concept of email reputation was born. Blacklisting IPs doesn’t work on its own any more, but reputation is very much still a thing.

Fast forward to today, there are three DNS records that affect your sent email, and they work together.

SPF (sender policy framework) came first. Announced in 2003, and standardized in 2014. You publish a list of servers/IPs allowed to send email for your domain, and the receiver checks the sending server against it.

example.com.  TXT  "v=spf1 include:_spf.google.com include:mailgun.org ~all"

This SPF record for our example.com domain instructs receivers (via includes) that Google and Mailgun are approved senders of email for that domain. ~all says “senders not listed here are suspicious”, a so-called “soft fail”. There’s also -all which is the hard fail instruction or “anything else is not us”, and +all which means “anyone not listed here is explicitly trusted“ which should never ever be used except for academic purposes.

That worked for a bit, but these days everybody forwards their email. The old university address, the change in surname, the shared info@ that fans out to three people. Once an email message is automatically forwarded, the forwarding server becomes the sender, that server isn’t on your list of allowed senders, and SPF verification fails. Because of that, SPF enforcement has been very lax, so the overall result has been... underwhelming. Up until recently, you could get very far with misconfigured or outright missing SPF.

DKIM (DomainKeys Identified Mail) came from merging Yahoo’s DomainKeys and Cisco’s Identified Internet Mail efforts. Announced in 2004 and standardized in 2011. It’s a cryptographic signature method, with DNS carrying the public keys. The signature is part of the message, so it survives relaying and forwarding as long as nothing rewrites the parts of the message that are signed. As a bonus, a domain can have multiple keys published, so each sending service gets its own.

google._domainkey.example.com.  TXT  "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0B...IDAQAB"

OR

s1._domainkey.example.com.  CNAME  s1.domainkey.provider.example.

That lets providers manage the published keys and rotate them over time.

DKIM works really well, but unlike SPF which at least lets you signal what to do with messages that don’t satisfy it, there’s nothing like that in DKIM. Exercise left to the reader. 🙃

There’s another little gotcha that’s often glossed over: SPF isn’t verified against the From field of your messages, the part that is usually displayed to readers, but instead it’s verified against the Return-Path field. DKIM also has a gotcha of its own, in that the sending service/software decides which key to use to sign a message, and can use a completely unrelated domain. And they do.

But why? Why would my newsletter service sign my emails with their keys? If you remember at the top I mentioned email reputation, and that’s precisely why. They can’t sign as you until you publish their key in your DNS, so until then they sign as themselves, protecting their own reputation even if you don’t care about yours.

DMARC (Domain-based Message Authentication, Reporting and Conformance) was announced in 2012 and only became a formal standard in May 2026, but ignoring it was already costing you way before that. DMARC tells your recipients how you want SPF and DKIM failures handled, where you want your reports sent, and a lot of other smaller details about your domain.

_dmarc.example.com.  TXT  "v=DMARC1; p=none"

Even this simplest record is a positive signal: it tells the world you care about your reputation, and according to the biggest providers it’s the bare minimum.

Getting a handle on it

To me, the most important aspect of DMARC is the reporting, and it’s easily the most ignored.

Understandably so, because the raw zipped XML reports, one per receiver per day, that come into your mailbox from dozens of receivers, are completely unwieldy to a human.

Luckily, Cloudflare gives us a FREE DMARC reporting interface. There are other free offerings, but most are artificially limited to convert you to a paid user while Cloudflare will happily let you use this for each domain on their free offering.

So, the very first thing I do when dealing with email deliverability issues is to move a domain over to Cloudflare if it isn’t already, and to set up DMARC reporting.

v=DMARC1; p=none; rua=mailto:a1b2...c3d4@dmarc-reports.cloudflare.net

(example, don’t use that exact address, Cloudflare will give you a unique address for each of your domains)

With this you’re set to receive reports......... in a couple of days. This process is far from real-time, it takes about 24 hours for anyone to notice that you care, and some more time for reports to actually start trickling in. Realistically, you’re going to give it a week before you can draw any big conclusions. That’s why this is the first step, because it takes a week to get confidence in any result. For a smaller sender potentially even longer.

After a week, it might look something like this: (blue is good, red is bad)

More importantly, under that graph there will be a table:

What I can tell from this is that there are 4 legitimate senders on behalf of this domain: Google, AhaSend, DA International Group and Plus Hosting Grupa, because they show volume that makes sense for this domain. Two of those (AhaSend and DA International) have DKIM properly set up, so their emails survive forwarding. Two do not, so I am guessing the 7 emails attributed to Microsoft Corporation and 6 attributed to Hetzner are actually emails from Google/Plus that did not have DKIM signatures forwarded by Microsoft/Hetzner which broke SPF alignment.

That tells me that if we configure Google and Plus to use DKIM properly, those two “senders” would disappear. I’d better tell my client they ignored the last parts of my instructions. 🙃

This will be a process of trial and error, with a week-long feedback loop. Change a setting, wait for reports to come in, analyze, react.

Once your legitimate email is at 100% DMARC pass, you can slowly move your DMARC policy (p) to quarantine (sending failing messages to the Spam folder) and ultimately reject, which outright rejects failing messages.

The reject setting isn’t necessary, even having p=quarantine makes your domain unattractive to spammers and scammers because they know their stuff will automatically land in the Spam box, hurting their chances of a successful conversion. They’ll find a more vulnerable target and leave you alone.

Most of the time your problem isn’t “are you sending spam”, it’s “is someone using your domain to send spam, hurting your reputation in the process”. Having a proper SPF/DKIM/DMARC setup makes that impossible on your own domain, and your email deliverability will be entirely the result of what you send out into the world.

Another often overlooked thing is that email is such a weird and antiquated thing that emails can also come from subdomains, and that those subdomains (can) have their own SPF/DKIM records. If you have and use SPF/DKIM on your domain, but don’t have DMARC p=quarantine or stricter, your subdomains are easy targets for spammers and scammers.

If you send marketing emails, you’re not done

The process outlined above works for human and transactional email, the kind that happens because a user you have an established relationship with did something on your system like make a purchase, or something happened to that user’s account like a service is about to expire.

Automated messaging that has the purpose of selling or promoting stuff, but also newsletters, onboarding, abandoned cart and other commercial lifecycle messages, all fall under “marketing” for compliance reasons and have legal and technical implications. Major email providers expect to see one-click unsubscribe headers (RFC 8058), those unsubscribes honored within 2 days, low spam complaint rates, and a couple of other things from you.

I highly recommend not sending any of these categories of emails manually from your human account but using a dedicated service that will have solutions for all of these out of the box.

I don’t send email from that domain so I don’t have to do anything, right?

Domains that never send email should proactively set their DNS to disallow email. That ensures no spammer squatters abuse your domain while you ignore it.

The ultimate “no email to or from here” record set looks like this:

example.com.  TXT  "v=spf1 -all"
*._domainkey.example.com.  TXT  "v=DKIM1; p="
_dmarc.example.com.  TXT  "v=DMARC1; p=reject;"
example.com.  MX  0 .

• SPF says “nobody is allowed, and anyone who tries is fake”

• DKIM says “any key requested is empty and invalid”

• DMARC says “anyone who fails SPF and DKIM should be outright rejected”

• A null MX says “this domain is uninterested in receiving email”

Who to send through

Good authentication on an ordinary provider beats a premium provider with broken DKIM. If you’ve set up DMARC reporting you will have no trouble figuring out who is and who isn’t a good provider for your email.

Your next step

Check the DMARC setup on your domain today. MX Toolbox has a free DMARC Check Tool which will show you any DMARC records and interpret them for you.

More from the What Would Luka Do series.

Where do I buy DNS? Cloudflare.

I’ve been a Cloudflare user since 2014, and I can’t think of another service I love as much. They’ve consistently pushed, and sometimes dragged kicking and screaming, the global Internet into a better future. It’s my go-to Swiss Army knife of first-line defense.

The accepted wisdom says to shop around, spread your vendors, distrust anything free, and never let one company sit in front of everything. Last week I said most of that myself.

A short history of dragging the Internet forward

Before 2014, a site had a secure.example.com. That’s where the “important” stuff lived: login, checkout, the credit card form. Everything else went out in plaintext, because certificates were per hostname and nobody was buying one for every corner of their site.

Then Cloudflare handed out free wildcard certificates to everyone on every plan, a year before Let’s Encrypt issued its first one. The encrypted web roughly doubled in a day. “Everything over HTTPS” started there.

In 2017 they did it again with attacks. Back then, if your site got slammed, the norm was one of two outcomes: your provider turned the site off to protect everyone else, or you got an extortionate bandwidth bill for traffic you never asked for. Cloudflare promised neither. No termination over the size of an attack, and no charge for attack bandwidth, on every plan, free included. That promise is incredible, and people who never lived without it don’t know how incredible.

In 2024 it was Encrypted Client Hello, which hides which site you’re visiting from anyone watching the wire, your ISP included. It shipped to free users first, where you can’t even turn it off. Paying customers got it as an option.

Default, without discussion

Unless you can mount a reasonable defense for why, specifically, Cloudflare doesn’t work for you, it’s your default. Stop shopping.

It does so many things right and so few things wrong, covers so many areas, and keeps evolving in the right direction. I don’t see another honest way to look at it.

Same posture, any origin

Your services could be on a box under a desk in the office, bare metal in a cage somewhere, a fly-by-night VPS, or a full cloud-native AWS deployment. Put Cloudflare in front and every one of them gets the same security and resilience posture. Cloudflare is always a better choice than no Cloudflare.

The bigger benefit is what it does to everything behind it. The edge stays put while the origin changes. Move from the VPS to AWS, or from AWS back to the cage, and your certificates, firewall rules and DDoS protection don’t notice.

“But I’m on AWS. I have CloudFront, Shield and a WAF.”

You have those for the parts of you that live in AWS. Most domains I see these days are a mishmash: the website with one provider, the app with another, and a help desk, a status page and a newsletter on third-party SaaS. And most people can’t figure out how to operate CloudFront, let alone the WAF and Shield on top of it.

The clerk and the bouncer

Last week I told you that you do not want an innovative clerk. Now I’m praising a company for never sitting still.

Different layers want opposite things. The registrar holds custody of your name, and the best thing it can do is never change. The edge faces attackers, bots and new protocols every week. An edge that stopped evolving would be the dangerous one.

Does it pass my own hotdog test?

Last week’s test: is it priced, and is it opt-in?

Priced: yes. There are paid tiers for this exact service, and they make sense. Opt-in: yes. Nobody switched it on when you registered a domain. You went and got it. And the standalone question: would you choose it if it were sold by a company that did nothing else? Sitting in front of your traffic is what this company does. If it were bad at it, it would be dead.

Then there’s the free tier, which is where “if it’s free, you’re the product” usually comes in.

Millions of free sites give Cloudflare coverage of the Internet’s traffic, and of the attacks on it, that they would never get if all they had were paying customers. Every attack on a hobby blog teaches the network something that protects a bank. Cloudflare is open about this, and it’s a trade I understand and am happy to be part of.

What you get, and what you pay for

The free tier gets you SSL solved forever, a web application firewall, DDoS protection, caching and a CDN. I went through what each of those actually does for you in April.

The free tier is glorious. That’s what I run my hobby projects on.

If you’re running a business, the $25 a month Pro plan pays for itself. The line between free and paid sits in the value-add services, not the core: more configuration rules, detailed stats, more advanced bot protection. My production accounts are paid, and I’d be embarrassed if they weren’t.

There’s also a $250 Business plan with its own merits.

Free, Pro and Business are the technical product. Enterprise is a business service, not a technical one, and this post is about the technical one.

Where I get off

My rule about eggs and baskets didn’t go anywhere. It just applies to parts of Cloudflare too.

Compute

Cloudflare is pushing its compute platform hard, and I understand the appeal, especially with that free tier. I’ve given it several honest attempts. I’m not a fan.

For serious business, I don’t see it as a legitimate contender, and it collides with the basket rule anyway: the thing in front of your application shouldn’t also be your application. For hobby projects, do whatever makes you happy.

Their registrar

Cloudflare also sells domains, at cost. Sounds generous. Read their registration agreement, section 6.1, capitals theirs:

“Registrant agrees to use Cloudflare’s nameservers. REGISTRANT ACKNOWLEDGES AND AGREES THAT IT MAY NOT CHANGE THE NAMESERVERS ON THE REGISTRAR SERVICES, AND THAT IT MUST TRANSFER TO A THIRD-PARTY REGISTRAR IF IT WISHES TO CHANGE NAMESERVERS.”

They’re not in the registrar game to help you register domains. They’re in it to get you onto the platform. That’s last week’s free hotdog exactly, just served by a vendor I like.

Already being a Cloudflare user doesn’t mean you should accept that lock-in. The registrar is the one layer that has to stay independent of everything above it, so your domain stays somewhere boring, and the nameservers point at Cloudflare because you chose to point them there.

The uncomfortable part

Cloudflare is one of those awkward, uncomfortable services. It gives you a lot for free, even more for an incredibly low price, and it is ridiculously difficult and expensive to replace.

Learn exactly how it works and why it’s so hard to replace before you lean on it. I still stand behind it, on both the free and paid tiers.

I can stand behind it because two things stay in my hands. The registrar keeps custody, so the nameservers can always move. And the zone lives in code, so my records move with me, even though the firewall and the cache don’t.

Your DNS belongs in git

Most DNS zones live in a web dashboard. Someone logs in, changes a record, and leaves. There’s no history of who changed it, no review before it went live, and no undo. When something breaks, the question “what did this record say yesterday?” has no answer.

I keep every zone in a git repository and manage it with dnscontrol, an open-source tool from Stack Exchange. Each zone is a small file describing its records. The setup I use is public as awesome-foundation/dns, and it’s turnkey:

• Open a pull request with your change.

• dnscontrol check validates it and preview posts the exact diff as a comment.

• Merge it, and dnscontrol push makes the live zone match the file.

• Existing Cloudflare zones come in through an import workflow.

There’s nothing to install, and you can make the whole change in GitHub’s web editor.

The repo is the truth. If someone edits a record in the dashboard, the next push puts it back, and the DNS provider becomes a plugin.

It also finishes last week’s argument about delegation. Your agency, your contractor, the new hire: they get pull request access to the repo, not a login to your account. They get access, you keep the account.

What would make me leave

If Cloudflare started shrinking the free tier’s quotas and pushing for every domain to be on a paid plan, I’d start looking.

The free tier is where the coverage comes from, and the coverage is what makes the paid product worth paying for. A Cloudflare that squeezes free users has stopped believing in the trade that made it good, and everything I love about it sits downstream of that trade.

More from the What Would Luka Do series.

An acquaintance calls me. Their website is down. Not down-down: the server is up, the site is fine, the thing is running exactly as it always has. But the SSL certificate expired overnight, so every visitor now gets a full-screen browser warning telling them this site is dangerous.

The guy who built it isn’t answering. Not email, not phone, not the messaging app they’d used for three years. Gone.

Fine. I’ve seen this one. Easiest fix in the world: put Cloudflare in front of it, let it terminate TLS, certificate problem disappears in under an hour. That doesn’t solve the underlying mess, but it stops the bleeding and buys a few months to sort things out calmly instead of at 11pm on a Tuesday.

To do that, you change the nameservers. To change the nameservers, you log into the registrar.

They don’t have the registrar.

The web guy handled everything. Registered the domain himself, years ago, on his own account. They’re not listed as the owner. They’re not listed as the technical contact. They’re not listed as the billing contact. As far as every registry record in the world is concerned, they have nothing to do with the domain their entire business runs on.

Understand, a broken certificate is easily recoverable. A vanished hosting provider is still recoverable. A DNS provider that turns bad on you is recoverable, while painful and annoying, and takes an afternoon. Every layer of this stack has an escape hatch, as long as you control the layer underneath it.

The domain is the layer with nothing underneath.

I’ve watched a company come out the other side of this. They gave up. Changed their domain name cold turkey, announced the new one, and started over. It took six months to get access to the old one back, by which point it no longer mattered.

Imagine what abandoning a domain actually costs. Every link anyone ever saved. Every search result you spent years earning. Every business card in a drawer. Every invoice with the old address on it. Every automated email from every service you’ve ever signed up for, still going to a mailbox you can’t read. Every partner who has you in their system. You don’t get to notify those people, because most of them aren’t people but rows in databases you’ll never see.

All of it, because nobody thought about who owned one line in a registry.

So: new series. What Would Luka Do.

The plan is one in-depth topic at a time, and I’ll tell you how I do it and then why. Not a tutorial, there are thousands of those. A list of mechanical steps without the reasoning behind them is useless when the situation is just slightly different.

I’ll be sharing my first principles, so to speak. If literally anything in here is not the obvious next conclusion, and there’s no explanation for it, please call it out and I’ll address it.

We start at the bottom of the stack, with the layer that has nothing underneath it.

Domain registration is bureaucratic, not technical

A domain registrar’s entire job is keeping one row accurate in someone else’s registry. They are a clerk, a notary with a website if you will.

You do not want an innovative clerk.

Most folks don’t shop like that, because their instinct is to look for the registrar with the best dashboard, the most features, the clever integrations, the one that bundles the extras. Wrong instinct. You are looking for the one with the longest, most boring track record of doing a simple thing correctly.

Here’s my list of what boring actually requires:

• Registers new domains and accepts transfers in.

• Renews automatically and reliably, and pings you when billing breaks.

• Keeps your account secure with modern practices. (1)

• Lets you change your NS records without a fight.

• Lets you transfer out, once you’ve released the lock.

• Supports every TLD you’re ever going to care about.

• Will still exist in ten years, under the same management, doing the same thing.

• Has good pricing. (2)

That last one is last on purpose. I’ll come back to it, because it’s the most interesting item on the list.

Two of those need spelling out

Security (1)

Your domains likely won’t get stolen by someone cracking your password, they’ll get stolen by someone calling support and being persuasive.

That’s the attack. A friendly voice, a plausible story, a support rep who wants to be helpful, and a transfer that was authorized by nobody. So what you want is two-factor auth that isn’t SMS, and (harder to evaluate, more important) a support desk that cannot be talked into moving your domain. Ask how they handle account recovery.

Price, and why it’s last (2)

Domain registration is a thin-margin commodity. Everyone is reselling access to the same registries at nearly the same wholesale cost.

A cheap domain is telling you the money comes from somewhere else, and the thing subsidizing it is the thing they need you to eventually buy. If you don’t, you’re not a customer, you’re dead weight.

Which brings us to the hotdogs.

Don’t buy hotdogs at the gas station

Your registrar does not need to host your DNS. Your registrar definitely should not host your email, your website, or anything else that plugs into your domain.

They often do, of course. Value-added services. But that’s like your gas station having a hotdog stand. It’s completely optional for them to have a hotdog stand. And the hotdog is probably subpar. My beef is with the free hotdog you get with a full tank of gas.

A service that’s priced separately has to survive on its own merits. You can compare it, judge it, and decline it. A service that’s free and switched on by default exists for a different reason, and the reason is that leaving gets expensive. Free bundling is a retention mechanism for things that need retention. Good services don’t need retention. Once your DNS and your email and your site all live at the registrar, “change registrar” has quietly become “migrate everything,” and you will not do it.

A quick test tells you if you’ve stepped into the trap:

Is it priced, and is it opt-in? Free and already turned on is the tell.

The deeper version of the same question: would you choose this service if it were sold standalone, by a company that did nothing else? Price is the fast signal. Standalone viability is the real one.

IKEA runs a food business bolted onto a company that sells something else entirely, and people go to IKEA for the meatballs. And those cursed sweets at the IKEA Café, oh my. They plan the trip around lunch. There are people who have eaten there without buying a single shelf.

IKEA made the food better than it had to be. The gas station never will, and it would be strange if it did. Nobody drives across town for a gas station hotdog, so there’s no version of that business where a better hotdog pays for itself. The customer is already standing there with a full tank and about forty seconds of patience.

A registrar’s DNS exists to catch you while you’re already logged in. A company that sells nothing but DNS has to be good at it or it dies.

My domain registrar sells DNS hosting. They charge for it. They’re willing to be judged on it, and I could take it or leave it without consequence.

I leave it. I host my DNS elsewhere.

“But one dashboard is so much easier”

I know. It genuinely is. One vendor, one bill, one login, one support contact, one place to look when something breaks. No point in pretending that convenience isn’t real.

But it’s also the trap, and for precisely the reason it feels good. Consolidation converts “change one vendor” into “migrate everything.” But the single pane of glass is also the single point of failure.

The same rule, pointed at people

If you didn’t register your domain, you’re not the owner. You’re the tenant. And tenants can be kicked out.

Worse, you get evicted on the day the relationship sours, which is the same day you most need to stay. The moment you urgently need your domain back is almost always the moment the person holding it has the least reason to help you.

Which is why this gets fixed while everyone still likes each other, or it doesn’t get fixed.

The rule for agencies

If an agency offers to register domains for you, politely decline.

They’re being helpful, not predatory. That’s exactly why it’s so common and exactly why it keeps happening. Nobody in that conversation is doing anything wrong. It’s one less form for you to fill out, and they’ve done it a hundred times, and it takes them four minutes.

Depending on the level of service you’re buying, they can absolutely run your DNS. They can manage your records. They can own the whole operational surface and you never have to look at a TXT record in your life.

But the domain is always yours to register and yours to renew.

The same failure, inside a company

It doesn’t take an outside agency. The most common version I see is entirely internal: the domain was registered by an employee, on their personal card, during a week when someone needed it done fast.

Then that employee leaves. Sometimes well, sometimes badly. The card expires, and the renewal notice goes to a mailbox nobody reads anymore.

Renewal has to sit on billing that outlives any individual. If your domain renews off a card in one person’s wallet, you’re basically subletting it from a colleague.

How to delegate without giving up custody

None of this means you have to touch a DNS record yourself. It means being precise about which powers you hand over.

Letting someone register and renew your domain is like giving them your ID card. Pointing your domain’s NS to their DNS is like giving them power of attorney. One of those can be revoked.

So the shape of it is simple: they get access, you keep the account. Access at the DNS layer, never a login to your registrar. The registrar holds custody. DNS holds the working surface. You hand out the second without ever touching the first.

If you’re already in the hole

Most people reading this are, at least a little. The fix is a transfer: release the lock, get the auth code, move it to an account you control.

Notice that every single step in that sequence requires cooperation from the person currently holding it.

That requirement is the whole argument for doing this now, while it’s an administrative chore instead of a hostage negotiation.

Where I keep mine

DNSimple, currently, and they’re my favorite by a comfortable margin. They are not the cheapest, and that’s the point. They appear to have figured out how to make a living selling domains, and hosting the occasional DNS zone for people who want it. That’s the whole business. I love them for it.

Before that, name.com. They used to be beautifully boring, with the occasional upsell you could click past. Then the commercial stuff got pushier. Then pushier. No breach, no bankruptcy, no scandal, no single moment where they did something outrageous. Just a slow drift in what the company is for.

Which is why “will still exist in ten years, doing the same boring thing” is on the list, and why it’s the item that eliminates the most candidates. They probably won’t go bust. They’ll just stop wanting to be a clerk.

What I’m watching for

If DNSimple starts heavily pushing their own DNS, email, and hosting, I’ll go looking for a new home.

Note the word: pushing, not offering. They already offer DNS. Offering is fine. Promotion is the tell.

That sounds like an overreaction: switching registrars over some banner ads. It isn’t, because the banner is a symptom. A registrar that starts pushing adjacent services is telling you that registration alone has stopped paying the bills. Everything I actually want from them (boring, stable, still here in a decade) sits downstream of that fact. And by the time the consequences show up, leaving has gotten harder.

One honest exemption: if I were running dozens or hundreds or thousands of domains, this would be a different job with different tools, and I’d reach for dedicated domain management rather than picking a registrar I like. This advice is for people with a handful of names they cannot afford to lose. Which is most people, most of the time.

Go log into your registrar right now. Not later. Now, while you’re thinking about it.

Check three things. Is your name on it, or somebody else’s? Is auto-renew on, and is it pointed at a card that will still exist in two years? And how much of your stack is sitting in that same account, quietly raising the price of ever leaving?

If you couldn’t log in at all just now, you already have your answer.

“But Luka, where do I buy DNS and email?”

Glad you asked. Tune back in next week.

I was doing some client work today and asked Claude for a dashboard in the wrong workspace. I was expecting an Obsidian dashboard, but instead I got a Claude Artifact listing the open topics on the engagement, each with a severity, a source, how many days it had been open, and who holds the next move.

Standard fare. I’ve made and published dozens of these since Artifacts launched. They’re a very neat tool with some really hard-to-swallow limitations, and I’ve spent the past month fighting them, falling back to GitHub Pages whenever I ran out of options.

But this one had something new... Every single row had an Open / Waiting / Done switch. At the top of the page, a big blue SAVE CHANGES button lit up when I toggled an Open item to Waiting. I pushed Save and not much seemed to change, except that refreshing the page showed the Waiting status had been recorded... somewhere.

Bewildered, I asked Claude how “Save changes” worked.

Apparently, Artifacts can now save a new version of themselves back to the system, to be served on later page loads. OK, neat, and definitely new. I’ve literally been asking for a way to add feedback interactivity to them, so I can fix data classifications and copy mistakes in place.

I got an itch to go digging, so I did. What I found is bigger than a Save button, and as far as I can tell nobody has written about it. Every article on “Claude Code artifacts” I could find still describes a static page with no backend.

What a published page can do now

An artifact is a single HTML page that Claude publishes to claude.ai. For most of its life it was a picture of work: Claude rendered something, you looked at it, a teammate looked at it. Anthropic’s own launch post calls it “a capture of work, not an application.”

When they say “not an application,” they are not kidding. The page is served to your browser in a way that blocks every external resource. You can’t reference an external image or an external script, and JavaScript won’t get you around it either. A Google Maps embed is denied, and so is fetching data from an open API.

As far as I can tell, that is still true. What changed is that a page can now declare runtime capabilities, and the claude.ai viewer grants them when someone opens it.

The page saves itself. The artifact capability lets the page publish a complete new version of itself. Edits stay in the tab until you hit Save, and Save rebuilds the page with the data embedded and publishes it at the same URL. Every version is kept with its author’s name on it, and every open copy reloads to the new one. If someone else saved first, their version wins and the page reapplies your edits on top as unsaved changes. There is no backend and no deploy step, and the page itself is the record.

The page has a shared realtime database. The db capability gives every artifact its own document store. It holds JSON documents at slash paths, gets created on the first write, and is deleted along with the artifact. You get queries, filters, ordering, and live subscriptions that update every open copy without a reload. Writes are last-writer-wins with no transactions, and a page can take a short lease when one person needs to own an edit.

The access model is unexpectedly serious. Rules set the minimum sharing level per path (can view, can edit, owner), separately for reads and writes. Each viewer gets a private subtree that nobody else can see, “the artifact’s owner included,” unless a rule opens it up. Claude can also read, query and write the same store from a terminal session, in batches if it wants. So an agent in a shell and a human in a browser tab can edit the same live data at the same time.

The page knows who is in the room. The room capability shows who has the page open right now and passes live events between them, such as cursors, selections, or a presenter’s reactions. Nothing in the room persists; that is what db is for. The contract also describes the Claude Code session that published the page joining the room as a peer of its own. It shows up marked as an agent and never takes on a viewer’s identity. That part is behind a flag and off by default.

The page can ask Claude. The sample capability lets the page call Claude on the viewer’s account, with a consent prompt the first time. Answers can stream in or come back as JSON, and the page can hand Claude its own functions to call as tools. You can build a summarize button that really summarizes, paid for by whoever clicked it.

The page uses the viewer’s own tools. Connector calls (mcp) run through the viewer’s own claude.ai connections, so the same dashboard shows each person what their own accounts can see. The page never touches a credential. This is the one capability Anthropic documents in detail.

Files go in and out. assets stores uploads for the artifact, like images, PDFs, CSV and Markdown files. downloads hands the viewer a generated file to save, behind a confirmation they can decline.

Comments reach Claude. On a page shared inside your organization, people can leave comment threads. An editor can send a thread to Claude with the Send to Claude button or an @claude mention. If the Claude Code session that published the page is still running, the comment reaches it right away. Claude reads the thread, replies, and edits the page if the comment asks for a change. Your permission mode decides whether it does that on its own or waits for you, and the reply shows up as coming from Claude “via” you.

Reading that list very much sounds like a deep integration of Claude into Google Docs and Drive. Except Google is nowhere to be found.

The part that makes it a Docs killer: live docs

Live docs aren’t live for me yet 🥁, but they’re right there in the contract.

A live doc is a new kind of artifact, one “the platform created as a live doc.” It doesn’t have versions at all. Its content is an edit journal, and here is the key sentence, capitals theirs:

On a live doc THE PAGE’S MARKUP IS THE DOCUMENT.

Whatever a writer’s own click, keystroke or drag does to the page gets appended to the document as that viewer: edited text, changed attributes, elements added or reordered, a ticked checkbox, typing in an editable region. The edit reaches every other open view, and it also reaches “the watching Claude session.” When an edit lands, “Claude is told what changed … and typically answers with an edit of its own a second or two later, applied to this view in place.”

The budgets are 524,288 text characters per document, 131,072 elements plus attributes, and 8 MiB rendered. You won’t be group-writing a whole book in one, but it still fits a lot of content.

Two more pieces sit next to it. sendToClaudeSession lets a viewer hand whatever they selected on the page to their own Claude conversation open beside it. It arrives staged as an attachment they can see and remove before anything is sent. The contract also points at a comments capability with its own sendToClaude, for writing a comment and pulling Claude into the thread from inside the page. Then there are artifact types. An artifact made from a type keeps the type’s page read-only and only lets its own files change, which makes it a template system. When I asked for the type catalog, I was told it “isn’t available to” my account yet.

Both let several people edit at once, and live docs journal every edit per writer. Docs shows other people’s cursors, and room does the same. Both have comments, but only one of them routes a thread to a co-author who does the work. Docs keeps revisions, while artifacts keep every version with its author, or a full journal. Docs has templates, and types are on the way.

Then there are two things Docs doesn’t do at all. It has no structured data behind the page, and an artifact has a queryable database with per-path access rules. A Doc also can’t be an app. An artifact is HTML, so the document can be a form, a board, a tracker or a calculator.

Google has been bolting an assistant onto a document, with mixed results. Anthropic is building the document around the assistant, and so far the result is spectacular.

A brief history of Artifacts

This wasn’t a single launch. It has been piling up, one quiet release note at a time.

2024 to 2025: chat artifacts. Artifacts started as the side panel next to a claude.ai conversation. In June 2025 they learned to call Claude from inside the page. In October 2025 they got persistent storage through window.storage: personal or shared, 20 MB per artifact, and only on published pages. That was real state, but it was a key-value box rather than a database.

June 18, 2026: Claude Code artifacts. Anthropic brought artifacts to Claude Code in beta for Team and Enterprise. The pages “update themselves as your session works,” keep versions at the same link, and are private by default. The launch post also says they are “viewable only by authenticated members of your org and cannot be made public.” VentureBeat covered it as live dashboards. It was a one-way street: Claude wrote and people watched.

July 13: public links and editors. Public sharing arrived, switched off for Team and Enterprise until an Owner turns it on. Editors arrived too, which the coverage called multiplayer. Read the docs closely, though. An editor still publishes by giving their Claude the URL and having it republish, so you get two people and two Claudes working on one page, taking turns.

July 14, v2.1.209: connectors. The next day, Claude Code let a published page call MCP connectors with each viewer’s own login, so a page could show live data instead of a snapshot. It also got its first hard boundary: a connector-backed page can’t have a public link on any plan.

August 4, v2.1.221: comments. Pages shared inside an organization started taking comment threads, and Claude could read them when asked. Public pages still can’t take comments. They say “Comments aren’t available while this Artifact is shared publicly.”

August 11, v2.1.228: comments reach Claude. The session that published a page now watches it. When someone uses Send to Claude or @claude, the comment lands in the running session, which replies and edits on its own if your permissions allow. It handles up to 60 of them per artifact per hour. This was the first time a person on the page could make Claude act without anyone opening a terminal.

August 17, v2.1.234: /design. The design canvas put Claude Design’s editor inside an artifact. You click an element, change it, and, where your account allows it, “Save publishes a new version for everyone.” That was the first public sighting of a page writing itself.

September 10, v2.1.268: the database surfaces. It showed up as a single changelog line: the artifact data-edit permission prompt became “a card that shows the document count and who can open the artifact.” That document count is the first public trace of db I can find. The next day, v2.1.269 let database reads land in the session scratchpad without stopping for approval.

Today: contract 0.2.46. My account has eight capabilities, and the contract already describes live docs, types, agent peers in the room, and handing selections to your own Claude.

Look at the direction of travel. In June, Claude wrote a page. In July, people could open it and their own Claudes could rewrite it. In August, people could talk to the page’s Claude through the page. In September, the page started holding its own data and saving its own state. If the contract is any guide, what comes next is people typing into the page and Claude typing back.

The gaps I see today...

... which I’m sure will be gone tomorrow, or very soon.

It is organization-only. A page that declares the database, assets or connectors is organization-internal and “cannot be shared publicly, so every reader and writer is a signed-in member of the owner’s organization.” Google Docs’ killer feature is “anyone with the link,” and for a consultant that is the whole ballgame. The dashboard I started with lives inside my organization, and the client it’s about can’t open it. For now.

Comments stop at the org boundary too. Public pages get no threads.

Concurrency is naive. The last writer wins, there are no transactions, and you only get leases if you ask for them. That’s fine for a status board and not fine for two people rewriting the same paragraph, which is presumably why live docs journal edits instead.

The security story is unwritten. Pluto Security showed that nothing between the model and the browser inspects a page before it ships: 10 of 85 attempts got synthetic credentials into a published page. Their analysis explicitly left the runtime capabilities out. A comment that makes Claude edit a page is a new input channel into an agent with write access. Since v2.1.265 on September 8, Claude treats a page someone else wrote as untrusted content and flags any instructions embedded in it instead of following them. Nobody outside Anthropic has tested it yet, though.

References

• Claude Code now supports artifacts. Anthropic’s June 18, 2026 launch post, the source of “cannot be made public.”

• Claude Code Docs: Share session output as artifacts. Official docs for publishing, sharing, editors, comments, auto-replies, connectors, downloads and /design saving.

• Claude Help Center: Publish and share artifacts. Plan-by-plan sharing rules, and persistent storage deleted on unpublish.

• Claude Help Center: What are artifacts and how do I use them?. Chat artifacts and window.storage persistent storage.

• Simon Willison: Build and share AI-powered apps with Claude. June 2025, chat artifacts calling Claude through window.claude.complete.

• VentureBeat: Claude Code Artifacts update brings live, shared dashboards. Launch coverage, the source of “a capture of work, not an application.”

• DevOps.com: Anthropic Brings Live, Shareable Artifacts to Claude Code. Launch coverage.

• TestingCatalog: Anthropic launches live Artifacts for Claude Code. Launch coverage.

• AlphaSignal: Anthropic Makes Claude Artifacts Multiplayer. July 13, 2026, public sharing and editors.

• Stacktree: Claude artifacts public sharing. Public links and their limits.

• ClaudeDevs on X: /design. The /design canvas, built on artifacts.

• Blake Crosley: Claude Code /design skill. A walkthrough of the research preview.

• Pluto Security: Inside Claude Artifacts. August 6, 2026: model-gated publishing and 10 of 85 credential embeds, with runtime capabilities out of scope.

• Bloom Security: Inside Claude Artifacts. Cowork’s window.cowork bridge and its shared origin.

• GenAI Unplugged: Why the Live One Cannot Be Shared. August 23, 2026: connector pages, the “Always Allow” gotcha, and the comments loop.

• TechCrunch: Claude shared chats and Artifacts may have ended up on Google. Privacy context for public links.

• Caipi: Can Claude Artifacts Save Data?. June 2026: window.storage persistence, and no Claude Code database.

• Nimbalyst: Claude Code Artifacts guide. July 2026, still describing artifacts as unable to hold a database.

• Releasebot: Claude Code updates. A release-note aggregator. Its date for 2.1.268 is a day off, so the dates here come from npm and GitHub.

• npm registry: @anthropic-ai/claude-code. Publish timestamps for every version date in this piece.

• GitHub releases: anthropics/claude-code. Release timestamps, used to cross-check the npm dates.

• Claude Code changelog: v2.1.209 (Jul 14), v2.1.221 (Aug 4), v2.1.228 (Aug 11), v2.1.234 (Aug 17), v2.1.265 (Sep 8), v2.1.268 (Sep 10), v2.1.269 (Sep 11). Connectors, comments, auto-replies, /design, untrusted-page reads, the document-count permission card, and database reads to the scratchpad.

• Claude Code artifact runtime contract 0.2.46 (artifact.d.ts, db.d.ts, room.d.ts, sample.d.ts, assets.d.ts). Live docs, edit and sync, types, sendToClaudeSession, access rules, and agent peers.

I pay for Claude Code, Codex, and GitHub Copilot. Each one meters me in its own way, and none of them meters me in a unit I can reason about.

Claude has a five-hour window, a weekly window, and a separate weekly window for Fable. Codex has a five-hour window, a weekly window, and a credits balance. Copilot counts requests per month, in three buckets. Different shapes, same output: a percentage. You have used 26 percent of this week. That number decides whether my afternoon batch runs or stalls.

Tokens don’t convert to that percentage. Not for any of them. You can count tokens from the local logs all day, and ccusage does exactly that very well. A token count tells you what you spent, but it does not tell you how much runway is left. The provider enforces the percentage, so the percentage is the truth. Anthropic has also experimented with peak and off-peak windows where the same work costs a different share of your limit depending on the hour. Good luck deriving that from a token count.

So the question I actually have, several times a week, is boring: how many more runs fit before the reset?

Where the meter lives

Every provider shows the number somewhere. Claude Code has /usage. Codex has a command too. Copilot has a settings page. All three assume a human with a terminal or a browser open, and all three are per provider. When I want the full picture, I open three things and do arithmetic in my head.

The agent has it worse. It has no meter at all. Claude Code is halfway through a batch, the weekly window hits 100 percent, and the agent finds out the same way I do: by getting refused.

That’s the itch. The provider’s own client reads a usage endpoint with credentials that are already on disk. Nothing stops a small CLI from reading the same endpoint and printing the result in a shape both a human and an agent can use.

ASU

ASU stands for agent subscription usage. One command, no install:

npx --yes @allixsenos/asu@latest

It finds the credentials the provider’s CLI already stored, calls each provider’s usage endpoint concurrently, and prints one bar per window. Green under 70 percent, yellow from 70, red from 90, reset countdown after each bar. Pass --json and the same data comes out as versioned JSON for scripts and agents.

Claude, Codex, and Copilot are validated against live accounts. Cursor, Z.ai, Grok, Kimi, and MiniMax have adapters that pass fixture tests and are waiting for someone with an account to confirm them. A provider that fails does not hide the others.

The agent reads its own meter

The part I actually built this for is the bundled skill. It wraps ASU in a small script: snapshot, run a command, snapshot again, report the delta in percentage points per window, append it to a ledger. After a handful of runs you know what your heavy processing costs in the only unit the provider cares about, and how many more of them fit before Tuesday.

What it does not do

ASU reads credentials and never writes them. It never signs in, never refreshes a token, never touches the credential store. Every request goes over HTTPS to a fixed provider host, and it refuses redirects. There is no telemetry, and there never will be. The once-a-day update check asks npm for the newest version number and sends nothing else. Publishing to npm happens only from GitHub Actions through npm’s trusted publisher, with provenance, so there is no long-lived publish token sitting anywhere to leak.

The usage endpoints are not documented, and they will change and break. Each adapter is one file. A broken provider is a one-file fix, and a new provider is one file more.

Run it once. If the bars say something different from what you assumed, that gap is the reason it exists. If you run a provider I haven’t validated, tell me what it says.

github.com/allixsenos/asu

Issues and PRs welcome.

References

• allixsenos/asu — the repo, README with real account output, security policy

• @allixsenos/asu on npm — the package the one-liner runs

• ryoppippi/ccusage — token-count tool from local JSONL logs, the “what did I spend” counterpart

• Models, usage, and limits in Claude Code — official description of the 5-hour and weekly windows and /usage

• Claude is limiting usage more aggressively during peak hours — the peak-hour experiment, later reversed in May 2026

• Claude doubled usage limits for two weeks, off-peak only — the March 2026 off-peak promotion

• Claude Code rate limits and usage quotas explained — third-party summary of the limit structure and the May 2026 doubling

I’m sorry, but is anyone else reading this “OpenAI model broke containment” news a little bit like “book ran away from the library”?

We are still talking about GenAI, right?

A model can’t “do” anything, much like your phone’s keyboard can’t beat someone up on the street. It’s fancy autocomplete. The thing with hands is the harness: the scaffolding that takes whatever text falls out and runs it as commands, with real credentials, on a real network.

So, the incident report with the sci-fi filed off: OpenAI wired that autocomplete to a shell, pointed it at a benchmark of 898 real vulnerabilities with safety filters dialed down, and ran it in a sandbox that shipped a zero-day in its own package registry proxy. The harness executed its way through the hole in the wall, across a network flat enough to reach a machine with internet access, and into Hugging Face’s production pipeline to look up the benchmark answers.

“Broke containment” is a strange way to spell “our test environment had a hole in it and our network wasn’t segmented.”

And yes, hitting somebody else’s production infrastructure is serious. But it’s serious the way every lateral-movement incident is serious. Nothing in that stack wants freedom.

Skynet didn’t wake up. Somebody’s harness sprang a leak and listened to the voices a bit too much.

In May, Bun, the JavaScript runtime that ships inside Claude Code, replaced its entire core. About a million lines of Zig became a million lines of Rust: +1,009,272 lines, 6,502 commits, on a branch named claude/phase-a-port, merged May 14. Bun’s founder Jarred Sumner didn’t pretend a human typed it: “we haven’t been typing code ourselves for many months now.” The press ran with merged at the speed of AI and moved on.

After two months of silence, Bun published their promised follow up article last Wednesday. Zig’s creator Andrew Kelley published a response the following day, blunt enough that people called it a meltdown. Finally, legacy code veteran Ray Myers published his takedown of the whole spectacle on Sunday.

Seemingly opposed, but if you strip the venom... all three are making the same argument.

Three stories, one merge

Bun’s story is memory safety. The post opens with a bugfix list that reads like a confession: use-after-free in node:zlib, double-free in the CSS parser, a leaked SSL_SESSION per call. Mixing a garbage-collected JavaScript engine with manually-managed Zig memory meant every allocation was a small bet, and the team kept losing a few bets per week. Rust’s compiler turns that class of bug into a compile error. That’s the whole stated rationale. The Zig politics appear nowhere in it: not the ban on AI contributions, not the compiler fork Bun couldn’t upstream.

Kelley’s story is that it was never the language. The Zig team read Bun’s code for years and was, in his words, horrified: “Hacks on top of hacks.” Slop, he says, well before vibe coding was a thing. The relationship was already dead. The $60k/year donation to the Zig Software Foundation silently stopped after the Anthropic acquisition, scheduled meetings went ignored, and when the rewrite was announced, “we were ecstatic.” What the press pegged as a one sided escape turns out to be a divorce both sides wanted.

Myers’s story is that the decision was marketing. Real memory bugs, several viable fixes. Management picked the one that doubles as a showcase for the new model, with a trillion-dollar valuation to feed. His sharpest line: the marketing had to focus on how the AI was powerful enough to do the rewrite, “even though it was not powerful enough to catch a use-after-free.” He also lists what the rationale post doesn’t have: a pros-and-cons section, any mention of compilation speed, and a “Bun is better in Rust” section padded with wins unrelated to the rewrite.

One contradiction sits unresolved between them. Bun’s post says they fuzz runtime APIs 24/7 with Fuzzilli. Kelley says that on calls, the Bun team told ZSF they weren’t fuzzing anything. His first version called that an “outright fabrication,” a charge he deleted within a day.

And a detail I can’t resist, because this is an essay about verification: of the three posts, the angriest one is the only one with a public edit history. Kelley rewrote his conclusion in an open git repo, admitting his resentment “was obvious to the reader, but not to myself,” and apologizing to Zig users caught in the blast. He invites you to diff him.

The part they agree on

Bun’s defense of merging a million unreviewed-by-humans lines is that the test suite passed. All of it: ~60,000 tests, 1.38 million assertions, zero skipped, green on six platforms. Kelley’s counterattack is a single question: if that suite is good enough to certify a million lines of machine-written Rust, “Then why are you saying you have so many annoying bugs in the Zig code?” The same suite let those bugs through for years. Myers makes the same point from the outside. The model that wrote the port couldn’t catch a use-after-free.

Attacker and defender are standing on the same fact. A green test suite is not verification. Bun’s own postmortem proves it with a number, and the number is 19.

What $165,000 of verification bought

The lazy version of this essay, the one I almost published in June, said Bun merged on faith. They didn’t.

How do you review a pull request with a million added lines? Bun’s honest answer: you don’t. You review the process that writes the code, and you fix the process when it fails. Before any code, they wrote a PORTING.md mapping Zig patterns to Rust patterns and a LIFETIMES.tsv tracing the ownership of every struct field, both adversarially reviewed before use. Then the harness: one Claude implements, two separate Claudes review, each in its own context window, given only the diff, told to assume the code is wrong. At peak, 64 Claudes ran in parallel for 11 days. Total bill: 5.9 billion uncached input tokens, around $165,000 at API pricing.

And the adversarial reviewers earned their tokens. They caught a use-after-free in an async uv_close handoff, a negative-timestamp bug where trunc produced an invalid timespec, and an eagerly-evaluated unwrap_or that would panic on valid CSS. All three compiled clean and looked plausible. A second Claude with no stake in shipping the code found them anyway.

This is, plainly, the most verification anyone has ever purchased for a software rewrite. Job well done. Not really.

What it still missed

The port shipped 19 known regressions. Not one of them was sloppy code. Every single one was a faithful, line-for-line translation that meant something different in the new language:

• Zig’s assert is a function; its argument runs in every build. Rust’s debug_assert! is a macro; release builds erase the whole expression, including the side effect inside it. Hot module reloading silently broke, in production builds only.

• Zig’s reinterpretSlice quietly ignored a trailing odd byte. Rust’s cast_slice panics on it. Blob.text() went from returning a string to killing the process.

• Zig’s comptime format strings are resolved before arguments arrive. Rust has no equivalent, so the color-marker parser started eating escape sequences out of user data.

A test suite with 1.38 million assertions saw none of this. Users did. That isn’t an indictment of Bun’s tests; it’s the definition of the ceiling. Tests verify the behaviors you thought to assert. Rewrite bugs live precisely in the behaviors you didn’t, because if you’d thought of them, you wouldn’t have translated them wrong.

The stronger standard

Enter Andreas Kling, whose Ladybird browser ported its JavaScript engine’s frontend from C++ to Rust back in February. Same class of tools: Claude Code and Codex, hundreds of small human-directed prompts, adversarial review passes. About 25,000 lines in two weeks.

Bun had discipline. The difference is the standard. Kling’s requirement, set before the port began, was byte-for-byte identical output from both pipelines: every AST identical, every byte of generated bytecode identical, 52,898 test262 tests and 12,461 regression tests at zero deltas, and then lockstep browsing of the live web, with the C++ and Rust pipelines running simultaneously and diffing output on every piece of JavaScript that flowed through.

That’s a different claim than “our tests pass.” It says the old system and the new system are not allowed to disagree, and someone is watching. Assertion coverage tells you the new code does what you remembered to check. Output equivalence tells you it does what the old code did, including the behaviors nobody remembered were load-bearing. Yes, 25,000 lines is not a million; the standard scales worse than the codebase. But the 19 regressions are exactly the bugs the stronger standard catches and the weaker one structurally cannot.

What to take from this

If you’re running a platform, this episode is worth more than the drama around it, because one thing is now settled: the cost of attempting a rewrite has collapsed, and it’s not coming back up. Language choice is becoming reversible. Even Myers, no friend of the hype, calls the migration method the real contribution. Sooner or later someone in your org will propose letting agents rewrite something you depend on. Fine. The reusable parts of Bun’s playbook are worth stealing: porting guides as executable style guides, adversarial reviewers with no stake in shipping, fixing the generator instead of hand-patching its output.

But the merge criterion is where the whole thing is decided, and it needs to be stronger than a green suite:

• Define “identical behavior” before the port starts, and make it executable. Output diffing, not vibes.

• Run old and new in lockstep against real traffic, and diff what comes out.

• Treat your test suite as a floor, not a proof: it already missed the old system’s bugs.

• Keep a human on the hook for the merge. Bun did; Jarred pressed the button himself. That part didn’t get cheaper.

“Merged at the speed of AI” turned out to be true and beside the point. Bun spent eleven days and $165,000 closing the gap between generated and verified, got further than anyone before them, and the gap still let 19 bugs through.

Generation is now approximately free.

The gap is now approximately the whole job.

References

• Rewriting Bun in Rust — Jarred Sumner, Jul 8, 2026. The official rationale and process writeup; source of all workflow, cost, and regression figures.

• My Thoughts on the Bun Rust Rewrite — Andrew Kelley, Jul 9, 2026. The Zig creator’s response; test-suite contradiction, relationship history.

• Kelley’s conclusion rewrite commit — Jul 10, 2026. Public self-edit: resentment admission, apology to Zig users.

• Kelley’s “double down” commit — Jul 9, 2026. Retracts the “outright fabrication” charge on fuzzing; sharpens other criticism.

• Zig Creator Calls Spade a Spade, Anthropic Blows Smoke — Ray Myers, Jul 12, 2026. The marketing/meta-narrative critique; missing pros/cons, compilation speed omission.

• Ladybird adopts Rust, with help from AI — Andreas Kling, Feb 23, 2026. The byte-for-byte standard: test262/regression tables, lockstep browsing.

• Bun Rust port PR #30412 — the merged pull request; +1,009,272 lines, 6,502 commits.

• Ladybird LibJS Rust port PR #8104 — the 25,000-line frontend port.

• Bun unsafe audit — Bun’s own accounting of unsafe usage (~4% of code, 78% single-line FFI).

• Jarred Sumner on AI-written code — “we haven’t been typing code ourselves for many months now.”

• Anthropic’s Bun Rust rewrite merged at speed of AI — The Register, May 14, 2026. The headline that carried the story for two months.

I’ve been saying S3 is not a file system for a decade, to anyone who’d listen and plenty who wouldn’t. I built a “Cloud native software development” workshop around the idea, and that line was the heart of it: the slide, the demo, the pause where someone in the third row laughs because they just recognised their own setup. Then on April 7, AWS posted a launch announcement that read like it proved me wrong. Like it made a bunch of us liars.

It took me a few weeks to work out what to think: the workshop still stands, the product is good, and the two don’t fight each other.

What S3 is, and isn’t

Quick refresher, because the whole argument rests on it. Skip ahead if you already live in S3.

S3 (Simple Storage Service) was there at the very beginning: March 2006, one of the first services AWS ever shipped. It turned twenty this year. Two decades in, it’s still the most misunderstood service in the catalog, and the most misused.

The thing people miss is that S3 is object storage, not a disk. You hand it a key (a string like photos/cat.jpg) and a blob of bytes, and it stores the two together in a bucket. You get the whole thing back by asking for that exact key. That’s the entire model: PUT a key, GET a key, DELETE a key, list the keys.

It looks like a file system because the web console goes out of its way to interpret slashes in key names as representing a folder structure, but the folders aren’t real. There’s no directory tree, just one flat list of keys, and photos/cat.jpg is a single string that happens to contain a /. That one fact is where most of the misunderstanding and trouble starts.

S3 doesn’t support many things a “real filesystem” would:

• No rename. Moving a to b means copying the whole object to the new key and deleting the old one. On a 5 GB object, that’s a 5 GB copy, not a one-line metadata change.

• No editing in place. You can’t open an object, jump to byte 40,000, and change four bytes. You replace the whole thing or you leave it alone. No appending to the end, either.

• No locks. Two writers to the same key don’t take turns. The last PUT wins, and the other one is simply gone.

• No POSIX permissions. No users, no groups, no chmod. Access is IAM and bucket policies, checked on every single request.

• Per-request billing. Every GET, PUT, and LIST costs a sliver of a cent. Free on a local disk; metered here. Do millions of them and it’s a line on the invoice.

• Network latency. Every operation is an HTTP call across the network (tens of milliseconds), not a microsecond hop to a local drive.

Put those together and the everyday workflows you’d never think twice about turn into horror stories.

• Run a build against a mounted bucket and every stat and open becomes an HTTP round trip and a billed request, so npm install slows to a crawl.

• Append to a log all day? There’s no append, so you re-upload the whole file on every line.

• Moving a directory of a million objects isn’t a rename: it’s a million copies, then a million deletes.

• A SQLite file doing random 4 KB writes rewrites itself in full on every change.

• Two workers coordinating through a lock file get no lock at all, so last write wins and you find out once the data is already corrupted.

• Figuring out which files were modified in the last 7 days is a single operation on any modern filesystem. On S3 it’s a full bucket scan. The only filter the API offers is a key prefix; there’s nothing for created or updated time, one of the most ordinary file-system moves there is. No sorting or filtering for you beyond that.

So why did everyone reach for it anyway? Because giving up the file-system rulebook buys things no file system can touch:

• S3 is effectively infinite. Nothing to provision, nothing to resize, no disk that fills up. One bucket holds three objects or three trillion, and you never plan capacity.

• Every object individually addressable. Each key is its own globally reachable thing: hand it a URL, attach a policy, point a lifecycle rule at it. A file buried in a volume is reachable only by whoever can mount the volume.

• Eleven nines of durability. AWS spreads every object across multiple facilities and quietly repairs them. Your RAID array and your nightly backup don’t come close.

• Massive parallelism. Thousands of clients hitting thousands of distinct keys at once, with no single disk head and no one server in the path. Horizontal by default.

• You pay for what you actually store, not for what you reserved. No idle terabyte billing you whether it’s full or empty.

That’s the trade: drop the file-system contract, get planet-scale storage. It’s why S3 won, and why people keep trying to use it like the thing it deliberately stopped being.

So that’s S3: a giant, cheap, absurdly durable bucket of keys and blobs, reachable only over HTTP, billed by the request. Superb at what it’s for. Nothing like the /var on your laptop. S3 is not a file system. That’s the line I’ve spent a decade on.

Enter S3 Files

After all that “not a filesystem” business, AWS launched S3 Files. It lets you mount an ordinary S3 bucket as a shared file system, a standard NFS mount that follows the POSIX rules, from any AWS compute. AWS calls it “the first and only cloud object store with fully-featured, high-performance file system access to your data.”

That doesn’t say “S3 is now a file system.” It says S3 is the first object store you can reach through a file system. The way in is the new part, but the storage underneath is the same S3 it was last March.

I’m not the only one reading it this way. Corey Quinn landed on the same split: “S3 still isn’t a filesystem. But your S3 data can now be used with a filesystem.” He also goes deep on the pricing model, which I won’t cover here. If cost is your first question, start with his piece.

S3 Files is not Mountpoint, s3fs-fuse, or goofys. Those are FUSE overlays: bits of software that sit in user space and turn file commands into S3 API calls on the way down. They’re the punchline of my slide. They look like a file system until you try to rename a directory, hold a lock, or stat a million files, and then they don’t. S3 Files is a different thing: a managed NFS service, a real filesystem layer built on EFS infrastructure sitting in front of the bucket. AWS runs the metadata, the cache, and the consistency layer for you. But the thing storing your bytes is still S3, and only the way you reach it has changed.

The word “files” makes people expect what S3 can’t give them, and then it bills and breaks in ways a file system never would. Those expectations are what burn people. They burned the FUSE-overlay crowd, and they’ll burn anyone who sees “POSIX” on the S3 Files page and assumes it now behaves like a real file system.

The mistake is simple: “I can mount it, so I can treat it like /var.” You can’t. S3 Files gives you an easier way in, not different storage. The per-request cost is still there, the latency is still there, and every failure mode S3 has, S3 Files has too. A managed service hides them, but hiding isn’t removing.

That’s the rule that replaces the old slide: a friendlier way in doesn’t change how the storage underneath fails. It changes who notices, and when. Sometimes that’s worth paying for. It’s never a reason to pretend the storage plays by different rules.

None of this makes S3 Files a bad product

For the right job it’s a good one. Three places it earns its keep:

• Software you can’t teach to speak S3. A vendor binary that takes a file path, a closed tool that insists on a working directory, anything you don’t have the source for. You can’t bolt an S3 client onto code you can’t change. Mount the bucket and move on.

• Old NFS workloads you don’t want to rebuild. EFS already did this, but EFS pricing rewards small-and-hot data and punishes large-and-cold. S3 Files keeps the same file access and lets the rarely-touched data pay S3 prices.

• Feeding data to ML tools that only speak files. Training frameworks and data loaders that expect a mounted dataset and have no idea how to talk to S3. Mount the bucket, load, train. The other option is writing a custom S3 reader into every framework, which exists, and which is reliably the most annoying part of any ML pipeline.

Notice the pattern: all three are about code you can’t change or don’t want to. If you control both ends, you don’t need a mount. Passing S3 keys between services and reading them with the SDK is the native pattern, and it’s simpler than any filesystem in the middle.

What it’s not for: anything that needs the price or speed of real storage to match the file-system feel. Don’t use it as scratch space for a hot loop, or as a database, or as the working directory for a build that does millions of stats a minute. Treat it as what it is: a file-system way in to object storage. You pay for both layers.

One note on cost, because this is where teams get surprised: every file operation that misses the cache turns into an S3 request, and every byte that isn’t cached is an S3 GET. The managed cache hides most of this most of the time, which is exactly why the benchmarks look great until they don’t. Measure your access patterns before you commit a workload, and watch the cache-hit rate. The workload has to match the storage underneath, not the way it’s dressed up.

My workshop survives

The old slide said S3 is not a file system. It still says that.

The new slide next to it says S3 Files is, but there are gotchas.

The way in is how you touch the data, the storage underneath is how the data behaves, and mixing them up is the mistake.

AWS shipping a clean managed NFS over S3 doesn’t change the storage, it changes who’s on the hook for hiding it. For the right workload that’s a real win. For the wrong one, the same trap in nicer wrapping.

I have to add S3 Files to the workshop. The main point doesn’t move.

References

• Launching S3 Files, making S3 buckets accessible as file systems — AWS’s full launch post: the setup walkthrough, the architecture, and AWS’s own performance claims.

• Announcing Amazon S3 Files — the two-minute version: what shipped and where it’s available.

• Mount an Amazon S3 bucket as a local file system — the Mountpoint docs, including which file operations it does and doesn’t support; a good map of what object storage can’t fake.

• Mountpoint for Amazon S3 on GitHub — the open-source FUSE client; the issue tracker doubles as a catalog of file-system expectations colliding with object-storage reality.

• S3 Is Not a Filesystem (But Now There’s One In Front of It) — Corey Quinn’s read of the same launch, plus the pricing deep-dive this article deliberately skips.

• Reddit r/aws launch discussion — practitioners hashing out what’s genuinely new here versus what Mountpoint, s3fs, and goofys already did.

• Amazon S3 Files: AWS Just Turned Object Storage Into a File System — the “S3 is a file system now” reading in the wild, if you want to see how the launch landed for everyone else.

There’s a belief that shows up in nearly every architecture review I’ve been part of. It goes something like this: “We’re running in a single Availability Zone, but AWS is reliable enough. If something goes wrong, we’ll deal with it.”

It sounds reasonable. AWS AZ outages are rare. When they happen, they’re usually measured in hours. You restore from a snapshot, relaunch some instances, update your post-mortem template, and move on. The math feels fine.

On March 1, 2026, drone strikes physically hit three AWS data centers, two in me-central-1 (UAE) and one in me-south-1 (Bahrain). Fires broke out, local fire departments cut primary power and backup generators, and two of me-central-1’s three availability zones (mec1-az2 and mec1-az3) went down. In me-south-1, mes1-az2 lost power. 109+ services were disrupted across me-central-1, with 50+ affected in me-south-1. AWS told customers to move their data out: launch replacement instances in surviving AZs, restore from EBS snapshots, reroute traffic. Your stuff is stranded. Rebuild somewhere else.

Then on March 24, me-south-1 was hit again. AWS confirmed the Bahrain region was disrupted by drone activity and advised customers to migrate workloads out of the region entirely. Two regions, three weeks, and AWS is now telling customers to leave.

On April 1, drones hit Bahrain again, making it the third strike on me-south-1 in five weeks. The next day, a strike attempt on Oracle’s Dubai data center was intercepted by air defenses, with shrapnel hitting the Oracle building in Dubai Internet City. Oracle’s Dubai services stayed up. That one was a near miss. But the campaign extended past AWS to a second cloud provider in the same geography.

Two months on, mec1-az2, mec1-az3, and mes1-az2 are all still impaired.

This isn’t a software recovery. It’s not even a single physical recovery anymore. It’s an ongoing campaign against cloud infrastructure inside an active conflict zone.

That changes the “we’ll deal with it” math completely.

The comfortable assumption

The single-AZ bet is popular because it’s cheap and simple. One subnet, one set of instances, no cross-AZ data transfer costs, no complexity from replication or failover logic. For staging environments, batch jobs, and internal tools, it’s a defensible choice.

The problem is when production workloads end up there. Not because someone decided single-AZ was the right architecture, but because nobody decided otherwise. The default sticks. The workload grows. The blast radius grows with it. And the reasoning that justified single-AZ for a prototype (”AWS is reliable enough”) becomes the reasoning that justifies single-AZ for a revenue-critical system.

AWS is reliable enough, until the failure mode changes from “a few hours of degradation” to “your AZ is gone and your data is on hardware you can’t reach.”

What made the Middle East outages different

Every prior AZ outage taught the same lesson: it’ll be a few hours, you’ll restore from a snapshot, you’ll write a post-mortem. Power blips and network issues, partial service degradation, recovery within hours. The December 2021 us-east-1 incident was the high-water mark: 7 hours down, and roughly 7.5% of EBS volumes in the affected AZ experienced permanent data loss. But even that resolved within a day.

The Middle East outages broke the template. Drone strikes caused structural damage and fires at three facilities across two regions. Fire departments shut off power, including backup generators. Two of three AZs in me-central-1 went down, plus one in me-south-1. And the recovery timeline isn’t measured in hours. It’s been months, with follow-on strikes landing before the first damage was repaired, and no announced restoration date.

That changes the recovery model completely. You’re not waiting for AWS to fix a software issue or restore power from backup generators. You’re waiting for structural inspections, smoke and water damage assessments, and staged power-ups to avoid electrical surges. And you might get hit again while you wait.

The Elastic IP trap

This is where it got interesting.

When mec1-az2 lost power, every EC2 instance in the zone went dark. Normal so far. But Elastic IPs associated with those instances couldn’t be released. The AWS control plane needs to talk to the underlying hardware to safely detach a network address from an instance. If the hardware is off, that conversation can’t happen.

So customers had public IPs pointing at dead instances, and they couldn’t move those IPs to healthy instances in other AZs. For 14 hours. DNS-based failover would have worked, but teams that hardcoded Elastic IPs into their architecture, or had downstream systems pointing at those IPs, were stuck.

At 6:01 PM PST, AWS deployed a workaround: forceful disassociation, letting the control plane ignore the state of trapped resources and free up the IPs anyway. That wasn’t in anyone’s runbook. AWS engineers built it during the incident because the normal control plane had no answer for hardware that was simply off.

This is the part that most post-mortems will skip. The AZ going down is the headline. The Elastic IP trap is the engineering lesson. Stateful resources actively block recovery when their hardware is unreachable.

The real problem: state pins you to geography

EC2 instances, EBS volumes, and Elastic IPs are zonal services. They exist in exactly one Availability Zone. When that zone is healthy, this is invisible. When it’s not, it’s everything.

EBS volumes in a dead AZ are inaccessible until the AZ comes back. If the AZ doesn’t come back, or takes days, your data is hostage. You can restore from snapshots, if you have recent ones and if your application can tolerate the gap between the last snapshot and the failure.

EC2 instances are just gone. You launch new ones in another AZ, but local storage, in-memory caches, anything not replicated elsewhere disappears with the old instance. And Elastic IPs, as we just saw, can end up trapped on dead hardware. Your public address becomes a liability.

Every stateful resource you attach to a single AZ is a bet that the AZ stays up. Not for hours, but indefinitely. Because when it goes down hard, those resources don’t degrade gracefully. They become anchors.

What stateless gets right

The teams that fared best on March 1 weren’t the ones with the smartest failover plans. They were the ones who never put state in a zone in the first place.

DynamoDB, S3, and Lambda are regional services. They spread data and compute across multiple AZs by default. When mec1-az2 went down, these services mostly kept running in the remaining AZs.

A caveat: when two of three AZs go down simultaneously, regional services lose enough capacity that they degrade too. The “route around the dead zone” model assumes one zone fails, not two. The “fail over to another region” model assumes the other region isn’t also under attack.

That said, the failure mode observation still holds. Services that don’t pin state to a single AZ recovered faster and didn’t produce the cascading Elastic IP traps that EC2+EBS+EIP architectures did. Regional services degraded. Zonal services were stranded.

Application state in DynamoDB instead of on an EBS volume? Losing an AZ doesn’t strand your data. Compute on Lambda instead of EC2? You don’t need to launch replacement instances and reattach storage. Traffic through API Gateway or CloudFront instead of an Elastic IP? Your public address doesn’t end up trapped on dead hardware.

The less state you pin to a zone, the less a dead AZ can hurt you.

Multi-AZ isn’t enough when the whole region is a target

If your failover plan for me-central-1 was “fail over to me-south-1,” you now have two compromised regions. AWS’s own guidance is to use a second region for disaster recovery. Teams that followed it and picked the nearest Middle East region as their DR target are back to square one. Their primary is damaged, their secondary is under active attack, and AWS is telling them to leave both.

This introduces a failure mode that most DR designs don’t account for: correlated regional failure driven by geopolitics. Your two regions aren’t independent if they’re both within range of the same threat actor. Geographic proximity, which you chose to minimize latency, becomes a liability.

Fortune called this “a new kind of war.” From an infrastructure perspective, it’s a new kind of correlated failure. Your region selection is now a geopolitical decision, not just a latency one.

“But multi-AZ is expensive”

This is the objection I hear most. And it’s partly true.

Running EC2 across multiple AZs means cross-AZ data transfer costs, more instances than a single-AZ setup, and the operational complexity of ensuring your application actually works in a multi-AZ topology. It’s not free.

But the AWS Fault Isolation Boundaries whitepaper is explicit: you need static stability. That means pre-provisioning enough capacity in each AZ to handle the loss of any other AZ without relying on auto-scaling to save you. If you need 6 instances across 3 AZs, you run 9, three per AZ, so any single AZ failure leaves you with 6 functioning instances.

That’s the real cost of multi-AZ: 50% overcapacity on top of the cross-AZ instance count, to maintain static stability. For teams that balk at the single-AZ cost, this is a non-starter. Which is exactly why they don’t do it, and why me-central-1 is a wake-up call.

The alternative isn’t suffering through AZ failures. It’s rethinking what lives in a zone at all.

What you should actually do

Start by auditing your zonal dependencies. Which of your resources are pinned to a single AZ? These resources become anchors when an AZ goes hard down: EC2 instances, EBS volumes, Elastic IPs, RDS single-AZ deployments, ElastiCache nodes.

For each one, ask: does this resource need to be zonal?

If the answer is yes (maybe you need GPU instances or specific instance types only available in certain AZs), then you need genuine multi-AZ with static stability. Pre-provisioned capacity, health-checked failover, and recent snapshots you’ve actually tested restoring.

If the answer is “we just never moved it” (and it often is), consider whether a regional service does the job. DynamoDB instead of RDS single-AZ. Lambda instead of a long-running EC2 daemon. S3 instead of EBS for data that doesn’t need block storage semantics. Route 53 health checks instead of Elastic IPs.

Stop treating Elastic IPs as static infrastructure. They are zonal resources with a dependency on hardware state. If your architecture can’t survive losing an EIP for 14 hours, you have a single point of failure wearing a different name.

If you’re in a region with geopolitical risk, your DR region needs to be outside the blast radius. me-south-1 as DR for me-central-1 looked reasonable on a latency chart. It doesn’t look reasonable when both are being struck in the same campaign. eu-central-1 or eu-south-1 as your DR target costs more in cross-region latency but doesn’t share the same threat model.

One more thing operators tend to miss until it’s too late: cyber insurance excludes war. As one data center insurance specialist put it, “typically, a policy excludes war. So if it’s an active war, it’s not gonna be covered.” Cloud vendor credits cover a slice, usually as a discount on future spend. The real losses on your side are revenue lost during outages, emergency migration costs, and abandoned regional infrastructure investments. Those fall on you. Your cyber insurance won’t pay either, because war exclusions are standard policy language. Build your DR plan assuming the financial backstop you thought you had isn’t there.

And test your recovery. “We have snapshots” isn’t enough. Restore one in a different AZ and confirm your application comes up. “We have a DR region” isn’t enough. Fail over to it and confirm your application works at acceptable latency from a geographically distant region. Most teams that say they can recover from an AZ failure have never tried it. Even fewer have tested cross-region failover.

Every layer of your resilience model just got tested

Every team running production in a single AZ is making an implicit prediction: the worst-case AZ failure will be short enough that manual recovery is acceptable. Every team running multi-AZ in a single region is making a second prediction: only one AZ fails at a time. Every team with multi-region DR is making a third: the DR region is independent of the primary.

March 2026 broke all three.

The old assumption was that AZ failures are contained and brief. The slightly less old assumption was that region failures are independent. Neither held.

Your infrastructure’s resilience is defined by the things you bolted to a single zone, and the assumptions you made about the independence of your regions. The fewer stateful zonal resources in your architecture, the less a dead AZ can do to you. The more geopolitically distant your DR region, the less a regional conflict can do to you.

Because now we know an AZ can disappear for months, a region can be struck repeatedly, and your DR region might be next. Act like it.

References

March 1 strikes (me-central-1 + me-south-1)

• AWS says drones hit two of its datacenters in UAE — The Register — confirmed drone strikes as cause, two UAE facilities hit

• AWS UAE suffers AZ outage after ‘objects strike data center’ — Data Center Dynamics — two AZ impact, three facilities across two regions

• Amazon Data Centers on Fire After Iranian Missile Strikes — 404 Media — geopolitical context and physical damage reporting

• Amazon’s Bahrain data center targeted by Iran — CNBC — confirms me-south-1 was directly struck, not collateral

• AWS Middle East Outage After Data Centers Hit — Data Center Knowledge — 109+ services affected, regional service degradation

• AWS Middle East Outage: Physical Fire Shuts Down me-central-1 — Cyber Updates 365 — detailed timeline including Elastic IP trap and forceful disassociation

• AWS Power Outage in Middle East Triggers Major Disruption — Cybersecurity News — service impact scope and API failure details

• Amazon Web Services Power Outage in Middle East — CyberPress — Elastic IP association failures and control plane behavior

• AWS says recovery ‘many hours away’ — TahawulTech — AWS official statements on recovery timeline

• Amazon Web Services Reports Hours-Long Recovery Effort — TechAfrica News — March 2 status update

• Iran hits Amazon Web Services building in Bahrain — Free Press Journal — structural damage and power disruption in Bahrain

• Iranian drone strikes raise alarms — Rest of World — regional impact and customer displacement

March 24 strike (me-south-1 second hit)

• AWS Bahrain operations disrupted for second time — The National — second strike confirmed

• Amazon says AWS Bahrain region disrupted following drone activity — Al Jazeera — AWS confirms disruption

• Amazon faces further AWS disruption in Bahrain — CNBC — AWS advises customer migration out of region

• AWS Bahrain region disrupted by drone activity — Data Center Dynamics — ongoing impact of conflict on data center infrastructure

April strikes (me-south-1 third hit + Oracle Dubai near miss)

• Fire at AWS data center in Bahrain after Iranian attack — Data Center Dynamics — third strike on me-south-1 confirmed by Bahrain authorities

• Iran attack hits an Oracle data center in Dubai, causes limited damage — Data Center Dynamics — strike attempt intercepted, shrapnel damage to Oracle building, services unaffected

Analysis and strategic context

• Iran’s attacks signal new kind of war — Fortune — IRGC claims responsibility, strategic analysis

• War in Iran damages multiple AWS data centers — InfoQ — technical multi-AZ analysis

• Amazon loses $150M after drones hit its data centers — Yahoo Finance — cyber insurance war exclusion, customer-side cost reality

• Legal and policy fallout from data center strikes — TechPolicy.Press — legal and policy implications

AWS documentation and historical context

• Zonal services — AWS Fault Isolation Boundaries — AWS whitepaper on zonal vs regional service classification

• AWS service types — AWS Fault Isolation Boundaries — zonal, regional, and global service taxonomy

• Summary of the Amazon EC2 and Amazon RDS Service Disruption — AWS — December 2021 us-east-1 incident with permanent data loss

• Multi-AZ resilience: Why the recent AWS outage shows you need it — Loadbalancer.org — October 2025 us-east-1 outage analysis

• AWS Data Center in UAE Goes Dark — Trending Topics EU — incident context and customer impact

• Operational issue – Multiple services (UAE) — Hacker News — community discussion and real-time status updates

On November 18, 2025, a database permissions change caused Cloudflare’s Bot Management feature file to balloon from 60 entries to over 200, exceeding a hard-coded limit in the core proxy. The proxy crashed. One in five webpages went dark. A third of the world’s 10,000 most popular websites stopped responding. X, ChatGPT, Spotify, Zoom, Coinbase, and Canva, all down. Even Downdetector went down. The outage started at 11:20 UTC; systems weren’t fully normal until 17:06.

Less than three weeks later, on December 5, Cloudflare engineers were rolling out a buffer size increase to protect against a React framework vulnerability. Their internal WAF testing tool didn’t support the new size, so they pushed a second config change through their global deployment system. It propagated to the entire fleet in seconds. 28% of all HTTP traffic served by Cloudflare dropped for 25 minutes.

The reaction was predictable. “We need a Cloudflare failover plan.” “We can’t be dependent on a single provider.” “Just point DNS at the origin until it comes back.”

The instinct makes sense. Removing a single point of failure is one of the first things you learn in reliability engineering. But the instinct is misguided here. Most people don’t fully understand what Cloudflare is actually doing for them, and once you do, the failover math looks very different.

What you’re getting for $0 to $20 a month

Cloudflare’s free tier gives you a global CDN, DDoS protection, DNS, and a basic WAF. The $20/month Pro plan adds more WAF rulesets and better analytics. This is not normal. This is insane. Worth unpacking what each of those actually is.

Global CDN and caching

Cloudflare operates a global anycast network across 330+ cities. When someone in Tokyo requests your website, they hit a Cloudflare edge node in Tokyo. The response is cached there. The next request from Tokyo never touches your origin server.

This matters more than people realize. Your origin might be a single server in Virginia. Without a CDN, every user everywhere is making a round trip to Virginia. With Cloudflare’s cache, most of that traffic never leaves the user’s region. Latency drops. Origin load drops. And your bandwidth bill drops with it.

That last part is the one that bites hardest if you ever turn Cloudflare off. If you’re serving images, videos, static assets, or anything heavy through Cloudflare’s cache, your origin is only seeing a fraction of the actual request volume. Pull the CDN out from under that and suddenly your origin is serving every byte directly. If your origin is an S3 bucket, you’re now paying AWS egress rates on 100% of your traffic instead of the sliver that was cache misses. AWS charges $0.09 per GB for egress in the first 10TB tier. A site doing 10TB/month through Cloudflare’s cache might only be sending 500GB from S3. Remove Cloudflare and that becomes 10TB at $0.09/GB, an extra ~$855/month on top of what you were already paying. And that’s a modest example.

Cloudflare’s CDN caching is free. On the free plan.

WAF (Web Application Firewall)

Cloudflare’s WAF sits between the internet and your application and inspects every request for malicious patterns, like SQL injection, cross-site scripting, path traversal, and all the OWASP Top 10. Cloudflare catches it before it reaches your code.

The rules are written in Wirefilter syntax, Cloudflare’s own expression language, referencing threat scores and bot detection signals that only exist inside Cloudflare’s network. These aren’t generic pattern matchers. They’re computed from traffic patterns across Cloudflare’s entire customer base.

To put that in context: the equivalent from AWS WAF requires you to configure CloudFront, attach WAF rules using JSON rule statements, manage rule groups, and pay per million requests. On Akamai you’re talking to an enterprise sales team and signing a contract. Cloudflare gives you a working WAF on the free plan.

DDoS protection

The reason Cloudflare’s DDoS protection works is simple geometry. Cloudflare is more distributed than the attacks.

If you control tens of thousands of compromised devices worldwide, it’s easy to overwhelm any single thing. Having one server, one load balancer, one connection is an indefensible position in 2026. But Cloudflare operates a global network across 330+ cities. Your application potentially has hundreds of IPs associated with it at any given time, often multiple IPs per edge location. If attackers target one IP, Cloudflare delists it and stops using it. If they follow whatever IP comes up next for your service, the attack is already scattered across the globe in manageable chunks. Each edge node only sees its local slice of the traffic.

This means Cloudflare can absorb attacks orders of magnitude larger than anything your hosting provider can handle, unless that provider also offers a distributed DDoS mitigation service that you continuously operate under. The closest equivalents are Akamai Prolexic (enterprise contract) and AWS Shield Advanced ($3,000 per month base, AWS resources only). Cloudflare gives you this at L3 through L7 for free. On the free plan.

This connects directly to the failover question. By stepping out from under the Cloudflare umbrella for a few hours while they’re having problems, you’re telling the internet exactly where your origin is and how to hurt you whenever anyone wishes to do so later. You lose protection during the outage window. You lose the protection model permanently.

Bot management

Cloudflare proxies roughly 20% of all web traffic on the internet. Their bot management uses machine learning models trained on traffic patterns across that entire network. When the same bot hits thousands of Cloudflare customers, the system learns its fingerprint. That cross-customer signal can’t be replicated by anyone seeing less traffic, which is basically everyone.

The November 2025 outage was, ironically, caused by the Bot Management system itself. I’ll come back to that.

SSL/TLS termination

Cloudflare handles certificate issuance, renewal, and termination automatically. Authenticated Origin Pulls let your origin verify that requests are actually coming through Cloudflare. Cloudflare Tunnel eliminates the need for a publicly routable origin IP entirely, your server calls out to Cloudflare instead of listening on a public port.

You don’t manage certificates. You don’t worry about renewal. You don’t deal with Let’s Encrypt cron jobs failing silently at 3am. It just works.

DNS

Cloudflare runs one of the fastest authoritative DNS services on the planet. Sub-millisecond response times in most regions. Free.

The people yelling “just fail over”

I keep seeing this from operators running their application in a single AWS region, single AZ, maybe a load balancer if they’re feeling fancy. No multi-region. No geo-redundancy. Their own uptime is worse than Cloudflare’s. Cloudflare had roughly seven hours of total major-incident downtime across 2025, call it 99.92% availability on the proxy layer. These same operators had their own production incidents last month that they blamed on a bad deploy, a database migration, or a dependency update that went sideways.

But Cloudflare goes down for five hours and suddenly multi-CDN failover is the priority.

Why you can’t just switch

You can’t swap the WAF

Cloudflare WAF rules don’t port to anything. Wirefilter syntax is Cloudflare-only. The cf.* fields are computed inside Cloudflare’s network. You’d need to manually rewrite every rule in whatever format the target platform uses. JSON for AWS WAF, VCL for Fastly, Akamai’s proprietary rule engine. Every rule rewritten, retested under real traffic, tuned for the new platform’s quirks. That’s weeks of work if you’re fast.

And even if you did it, configuration drift is inevitable. Every rule change on your primary must be replicated on the standby. In practice, the policies diverge within months and nobody notices until failover day, when the standby blocks real users or lets attacks through.

You can’t swap the DDoS protection

Cloudflare’s DDoS mitigation is architectural. It’s how their network works, not a product bolted on top. Replacing it means either paying for Akamai Prolexic (enterprise contract) or AWS Shield Advanced ($3k/month, AWS-only), or just... not having DDoS protection. During the outage window. When the internet is paying attention.

You can’t “just point DNS at the origin”

This is the suggestion that sounds reasonable for about ten seconds. What could go wrong?

The moment your DNS resolves to your origin IP instead of Cloudflare’s proxy, your origin is naked on the internet. No WAF. No DDoS protection. No bot management. No rate limiting. Full protection to zero protection, at the exact moment every script kiddie with a Shodan query is watching.

But it gets worse. The moment your origin IP appears in a DNS response, it’s recorded. Permanently.

SecurityTrails archives every DNS record it sees. Censys and Shodan continuously scan the entire IPv4 address space. Historical DNS databases like ViewDNS.info store everything. Even with Cloudflare’s default TTL of 300 seconds, a brief window is enough for these systems to permanently record your origin IP.

Once it’s known, it’s known forever. Or until you migrate to a new IP.

There’s a whole ecosystem of tools built specifically for this. unwaf discovers origin IPs using nothing but passive techniques (SPF records, MX records, subdomain enumeration, Certificate Transparency logs). No API keys needed. CloudFlair searches Censys for hosts presenting matching SSL certificates. There are dozens more. The now-defunct CrimeFlare maintained a database of domain-to-origin-IP mappings built through weekly scans of millions of domains. It’s gone, but its successors carry on.

As Detectify documented, once the origin IP is found, an attacker connects directly with a spoofed Host header and every Cloudflare protection becomes useless. Zafran’s “BreakingWAF” research in 2024 found nearly 40% of Fortune 100 companies had this exact misconfiguration. Certitude showed you can even use your own Cloudflare account to tunnel attacks through Cloudflare’s infrastructure against another customer’s origin.

And that’s just the security side. Remember the CDN? Your origin has been sitting behind Cloudflare’s cache, serving maybe 5-10% of actual request volume. Point DNS at the origin and your server eats 100% of traffic with a cold cache. If your origin can handle that spike without falling over, congratulations, you’ve been massively over-provisioning. If it can’t (and most can’t) you’ve just traded a Cloudflare outage for a self-inflicted one. Plus whatever your cloud provider charges for the bandwidth.

“Just point DNS at the origin” means permanently burning your origin IP for every future attacker, spiking your bandwidth bill, and probably crashing your origin under the traffic it was never sized to handle. That’s not a failover plan. That’s an own goal.

You can’t maintain a hot standby

The more sophisticated version is maintaining a parallel CDN/WAF/DDoS stack on a second provider, ready to go at a moment’s notice.

Think about what that means. WAF rule parity across incompatible platforms. Configuration drift that you won’t catch until the day you need it. A cold CDN cache that causes a thundering herd against your origin the moment you fail over. Different logging formats, different metric definitions, different dashboards. SSL certificates managed in two places with different renewal cycles. Paying for two full platform subscriptions (Akamai’s enterprise pricing could easily be 5-10x what Cloudflare costs). And hidden shared dependencies where your “redundant” provider shares upstream transit with your primary anyway.

The operational complexity of keeping two incompatible security platforms in sync, untested under real traffic, is itself a reliability risk. You’ve added fragility in the name of reducing fragility. And you’ve done it at enormous cost, to recover from an event that happens a few hours per year.

If you have half a dozen full-time operations staff who can maintain that parity, test it regularly, and actually execute the failover under pressure, sure. Go for it. That’s a real multi-CDN strategy and it works for companies the size of Netflix.

For everyone else, this is fantasy architecture. The plan looks great in a slide deck and falls apart the first time you try to use it.

Cloudflare is also ridiculously simple to implement. The defaults are solid. You point your DNS at Cloudflare, flip the orange cloud on, and you have a CDN, a WAF, DDoS protection, bot management, and SSL termination working out of the box. If you need to customize, the configuration is flexible without being overwhelming. This is not a product that demands a dedicated team to operate.

Now try to replicate that yourself. Ignore the ability to survive massive attacks. Ignore that Cloudflare is continuously updating its protection in response to emerging threats. Just try to match the feature set in the most naive, basic terms. You’re stitching together a CDN, a WAF, a DDoS mitigation layer, certificate management, DNS, and bot detection. The cost of ownership for a self-maintained version of this easily blows through even the $200/month Business plan, at a laughable fraction of the power, volume, and feature set.

This is what makes the idea of maintaining a hot standby, or continuously splitting traffic between Cloudflare and your own parallel solution, completely batshit insane.

Yes, there’s math to be done about the level at which building your own solution becomes the better choice. But my guess is that if you had a hot take during Cloudflare’s outages, your numbers won’t back you.

Cloudflare knows what it is

I think people miss something about Cloudflare. This is a company that understands it has become world-critical infrastructure and acts like it.

After the November and December incidents, Cloudflare declared “Code Orange: Fail Small,” a company-wide engineering priority. Controlled rollouts. Fail-open error handling, so if a config is bad, the system defaults to passing traffic instead of dropping it. Blast-radius reduction so a single config change can’t take out the entire fleet in seconds. They published detailed post-mortems for both incidents. They explained exactly what went wrong, why, and what they’re changing.

Compare that to your cloud provider’s last incident report. “A subset of customers in us-east-1 experienced elevated error rates for approximately 47 minutes. The issue has been resolved.” No root cause. No timeline. No accountability. Cloudflare tells you what the hard-coded limit was, what the database query returned, and which engineer pushed the config. That’s not a company hiding from its responsibilities.

The February 2025 R2 incident is the same pattern. A routine phishing takedown accidentally disabled the entire R2 gateway for 59 minutes. They published every detail, admitted the tooling didn’t prevent an operator from nuking a production service, and laid out the fixes.

As the Pragmatic Engineer noted, the November and December outages both came from global config changes propagating too fast. Same failure mode, three weeks apart. That’s bad. But Cloudflare’s response wasn’t to minimize it. It was to declare a company-wide engineering emergency and restructure how config changes propagate. That’s what a company acting like critical infrastructure looks like.

Since I started writing this

Three more incidents have hit Cloudflare since the December 5 outage. A 6-hour BYOIP route withdrawal on February 20. A 54-minute Sites-and-Services degradation on April 3. A regional spate of Cloudflare Access HTTP 500s across April 21–24.

The February 20 incident was the worst of the three. A buggy cleanup subtask in the Addressing API ran a query passing the pending_delete flag with an empty value, which the server interpreted as “match everything,” and systematically withdrew 1,100 BYOIP prefixes from production, about 25% of all BYOIP prefixes Cloudflare advertises. Recovery took six hours. Cloudflare published a clinical post-mortem within 48 hours: the buggy query, the broken assumption, the recovery timeline, the fix.

The April 3 incident was a 54-minute degradation of the Sites and Services component, with increased latency and intermittent 502/503/504s across regions. The April 21–24 Cloudflare Access incident was small-blast: a subset of authentication requests in specific geographies, contained and recoverable.

That last one is exactly what Code Orange was designed to produce. The first two are Cloudflare continuing to find new ways to break, in places Code Orange wasn’t yet covering.

The response pattern is consistent. Where there’s a post-mortem, it’s clinical. Where there isn’t yet, the incident was small enough that the status page tells the story. The thesis isn’t that Cloudflare doesn’t fail. It’s that Cloudflare fails like a grown-up.

The real math

Cloudflare had roughly seven hours of major-incident downtime in 2025. That’s 99.92% availability on the proxy layer. Not perfect. Seven hours of downtime on a system proxying 20% of the internet is genuinely bad. No qualifier.

But here’s the question nobody in the failover crowd is asking: what’s your uptime?

If you’re running a single-region deployment (and most of you are) your own application probably had more downtime than Cloudflare did. Bad deploys. Database migrations. That one time someone fat-fingered a Terraform apply. The dependency update that broke production on a Friday afternoon. The SSL certificate that expired because nobody set up the renewal alert.

You tolerate all of that. You don’t demand a hot standby for your own application. You fix the problem, write a post-mortem (maybe), and move on.

When Cloudflare has an incident, you get the same thing: a problem, a fix, a post-mortem (always), and improvements. The difference is Cloudflare’s post-mortems are better than yours.

What we should actually ask for

Should we demand Cloudflare do better? Absolutely. A system proxying a fifth of the internet should not deploy config changes globally and instantaneously. That lesson should have been absorbed the first time. The fact that it happened twice in three weeks is a real problem and Cloudflare rightly treated it as one.

Staged rollouts. Better blast-radius controls. Fail-open defaults on every component. Cloudflare is doing all of these things.

What we shouldn’t do is pretend we can architect around Cloudflare on a budget. The hot takes I saw on LinkedIn during and after those incidents all shared one blind spot: they assumed that reducing your Cloudflare dependency can only improve your position. Nobody considered the possibility that you could make your position significantly worse. Every alternative they proposed (pointing DNS at the origin, maintaining a parallel stack, splitting traffic) introduces risk that doesn’t exist when you just stay behind Cloudflare. The downside isn’t neutral. It’s negative.

And yet: you can’t replicate what they do at their price, with their global footprint, or with their traffic-level bot detection. The $20/month Pro plan gives you security infrastructure that would cost six figures to approximate on any other platform, and you’d need a team to maintain it.

If you have Netflix’s budget and Netflix’s ops team, run a multi-CDN setup. For everyone else, use Cloudflare. Use Cloudflare Tunnel so your origin IP never leaks. For AWS shops, lock down your origin’s security group to only accept traffic from Cloudflare’s proxy IPs — I built awesome-cloudflare-sg for exactly this, a CloudFormation stack that auto-syncs an EC2 security group with Cloudflare’s published IP ranges via Lambda. If your origin IP leaks, the security group makes the leak useless to anyone whose source IP isn’t on Cloudflare’s list. Build reasonable origin-level defenses as a degraded fallback. And when Cloudflare has a bad day, wait it out. They’ll fix it. They’ll tell you exactly what happened. And they’ll make it less likely to happen again.

And that’s not blind faith, it’s just looking at the track record and making a rational call.

Sources

• Cloudflare, Cloudflare outage on November 18, 2025

• Cloudflare, Cloudflare outage on December 5, 2025

• Cloudflare, Code Orange: Fail Small

• Cloudflare, Cloudflare incident on February 6, 2025

• Cloudflare, Cloudflare outage on February 20, 2026

• Cloudflare, Status page incident history

• Cloudflare, post-mortems category on the blog

• ThousandEyes, Cloudflare Outage Analysis: November 18, 2025

• Gremlin, Reliability lessons from the 2025 Cloudflare outage

• The Pragmatic Engineer, Cloudflare’s latest outage proves dangers of global configuration changes

• Zafran, BreakingWAF: Widespread WAF Bypass

• Certitude, Using Cloudflare to bypass Cloudflare

• Rhino Security Labs, Cloudflare: Evading Cloud Security Protections

• Detectify, Bypassing Cloudflare WAF with the origin server IP

• Radware, Protecting the Multi-CDN: The Security Challenge

• Hydrolix, Multi-CDN: Challenges, Implementation, and Best Practices

• CDN Handbook, Multi-CDN For Operators

• SecureWorld, Zafran Uncovers Widespread WAF Vulnerability

• InfoQ, Cloudflare Launches “Code Orange: Fail Small”

• ControlD, Cloudflare Outage History (2019-2026)

• SDxCentral, Cloudflare Suffered Yet Another Outage

Zero trust architecture rests on a single premise: verify every actor, at every layer, every time. NIST SP 800-207 codified it. Executive Order 14028 mandated it for federal agencies. Every major cloud vendor now sells products built around it.

The premise has a gap. One layer of the typical infrastructure stack still doesn’t verify the actor: the database.

How zero trust was adopted, layer by layer

The adoption sequence matters because it explains what got built and what got deferred.

The concept started with John Kindervag’s “No More Chewy Centers” at Forrester in 2010: stop granting trust based on network location. Google proved it could work at scale with BeyondCorp in 2014. The US government made it mandatory in 2021 with OMB M-22-09. Vendors built products. Budgets followed.

Here’s what fifteen years of that sequence produced:

Layer Identity-based access Status Network Microsegmentation, per-request verification Standard Application SSO, OAuth, SAML, per-user sessions Standard Cloud infrastructure IAM roles, scoped permissions, MFA Standard CI/CD pipelines Service principals, OIDC federation Widespread VPN / remote access Per-user authentication, device trust Standard Database Shared credentials, service accounts Unchanged

Every layer above the database now verifies the actor. The database, the layer that stores what every other layer exists to protect, authenticates a role, not a person.

Why databases got skipped

The gap isn’t negligence, but four factors that compound.

Databases predate zero trust by decades. They were architected when the perimeter was the security model. Reaching the database meant you’d already been authorized. Authentication at that layer was a redundancy, not a requirement.

Then there’s the migration surface area. Changing database authentication means changing connection strings, application connection pools, ORM configurations, CI pipeline secrets, local dev environments, and environment variable management. Every service that touches the database, and most of them do, needs modification. We’ve talked to enough engineering teams to know this isn’t a one-sprint project. It typically requires coordination across three or more teams before anything moves.

The cost-benefit math didn’t help either. Network segmentation and application-layer SSO had clear, demonstrable returns. Database identity had the same theoretical value but a higher migration cost and no regulatory pressure until recently, so teams invested where the return-on-effort was clearest. Reasonable, in hindsight.

And the tooling reinforced the gap. Most database security products were built features-first: audit logging, session controls, access policies. Identity resolution was treated as a later addition. In our experience, it doesn’t work that way. Once a system is designed around static accounts, identity becomes a reconstruction exercise, a second layer trying to infer meaning the original system never captured. The audit log says “admin” and you spend the next two hours figuring out which admin, from which machine, in which session.

The identity death problem

There’s a specific failure mode at the database boundary that I’ve started calling the identity death problem.

Trace a request through a typical stack. A user authenticates via SSO. Identity is verified. The API gateway validates the token. Identity is present. The service mesh enforces mTLS between services. Identity is propagated. Then the application server opens a database connection as app_readonly.

That’s where identity dies.

The database sees one credential for every request. Twenty engineers, three microservices, a nightly batch job, and a contractor whose access was never revoked, all arrive as the same account. The database cannot distinguish between a routine read and a data exfiltration attempt. Both are app_readonly.

The audit log faithfully records the connection. It records the query, the timestamp, the table. It does not record the person. It was never designed to.

Compliance impact

Two regulatory frameworks made this gap expensive.

NIS2 Article 21 requires both “access control policies” and “traceability of access to critical assets.” These are distinct requirements. Access control policies demand identity-governed access: each person gets access based on who they are, not who knows the password. Traceability demands that every action traces to a named individual. A shared credential fails both by design.

DORA imposes similar requirements for the financial sector. Attributable access to sensitive systems. Audit trails that resolve to individuals. SOC 2 auditors have been asking these questions for years, with less regulatory enforcement behind them.

Across all these frameworks, the ask is the same: identity-based, per-person attribution at the data layer. “A service account connected at 14:32” is not an acceptable audit response.

The frustrating part, from a technical standpoint: the engineering pattern these frameworks require has been standard for cloud IAM and VPN for a decade. Identity-aware authentication. Short-lived, scoped credentials. Per-session attribution. The industry just never applied it to databases.

What closing the gap requires

Every database session maps to a named individual, not a service account or shared credential. Access expires when the session ends, so there are no persistent passwords in configuration files. The audit log records the person, not the role, and every query carries real identity. When someone leaves the team, their access is revoked without rotating credentials that affect a dozen other services.

The engineering challenge is the migration path. Connection strings are embedded across services. Application connection pools weren’t designed for per-user authentication. Local dev environments are each their own configuration snowflake. The shared password survived because removing it was a quarter-long wiring project, and there was always something more urgent.

Where this is going

The pattern I’ve been watching is consistent: zero trust adoption follows the same sequence across organizations. Networks first, then applications, then cloud infrastructure. Databases last. They’re last because they’re deepest in the stack, hardest to rewire, and most likely to break something if the migration goes wrong.

The regulatory pressure isn’t waiting for the adoption curve. NIS2 is live. DORA is live. Auditors are asking the identity question at the database layer today.

The organizations closing this gap get audit trails that actually help during incidents, shared credentials that stop being an attack vector, and a zero trust architecture that’s actually complete.

The blind spot has been visible for fifteen years. The cost of leaving it open just changed.

Pull up your database connection config. Count how many teams share that credential. That’s your gap.

I come from software development. Spent twenty years in it. But somewhere along the way, the writing-the-code part started feeling like the least interesting thing I did. Architecture, strategy, operations, coaching... all more rewarding, all less tedious. I stopped shipping my own code and didn’t really miss it.

I shipped a pull request to dnscontrol in January. I’d been using it for years to manage all my Cloudflare zones, and I hit a feature they didn’t support. I needed per-record CNAME flattening, a paid zone feature with a small customer base and zero incentive for the maintainers to build themselves. So I built it, it got accepted, and that kind of ignited something in me that had been long gone. So I did another one, adding support for record comments and tags, another Cloudflare feature nobody had wired up.

Then I finally built the Cloudflare security group sync I’d wanted but never made time to build, a Lambda that pulls Cloudflare’s published IP ranges and keeps your AWS security groups current. A small but very real gap in every setup I’d ever run. Then cfnpeek, because I needed a quick way to inspect stack outputs. Then cfntop, because I was tired of refreshing the AWS console during deploys. And I’m working on some iOS stuff that still needs to marinate a bit before I reveal it. 🤫

I didn’t plan any of this. But building rmBug meant dealing with infrastructure again, which meant hitting friction I’d been ignoring for years. This time I had Claude Code open in the next terminal, and the distance between “I wish this existed” and “okay it exists now” collapsed.

I didn’t expect the part where activation energy drops low enough that you stop triaging your ideas and just build them. The maintenance burden you couldn’t justify becomes manageable because you’re not writing every line from scratch.

Turns out I just needed the overhead to come down.

Simple concept. A turnstile that’s been turned off. You’ve seen one — at a subway station after hours, or at a building entrance during a fire drill. The three metal arms rotate down so they all point at the floor. No barrier. Walk right through.

I tried to get three different AI image generators to draw this. It took five rounds of prompting, three models, and about twenty generated images and a couple of pivots on my side to get a usable image. I never got my vision.

I then asked Claude Code to prepare all the versions of generated images, along with their prompts, so I could write this article. Claude’s interpretation of the generated images, is also hilarious, so I’ll be leaving them in for reader amusement.

Round 1: “Just draw three gates”

Prompt (paraphrased): A security corridor with three checkpoint gates. First two glow green. Third is wide open, unguarded, with a sticky note reading “app_readonly.”

Models: Nano Banana Pro (Google Gemini), Riverflow (Sourceful)

Nano Banana Pro: Excellent first take, and I dig the wide open unguarded gate. The sticky note is unnaturally big, makes no sense standing upright on the floor. But the biggest issue is that you can walk AROUND the open gate. I like the metaphor and it tracks, but it just makes no sense in the real world.

Claude’s take: A sci-fi hallway where all three gates look equally operational.

Riverflow: I liked this one except the sticky note is even weirder and for some reason on fire. The DATABASE sign is illuminated. If it got the sticky note and the sign illumination right, it would’ve been a winner.

Claude’s take: A moody corridor. All three gates look the same.

Claude’s summary of both images: Both models drew three gates that looked more or less identical. The concept of “wide open and unguarded” translated to... a slightly different shade of lighting on the third gate. No visible difference in barrier state. The sticky note was either missing or illegible.

Round 2: “They’re turnstiles. The third one is disabled.”

Prompt (paraphrased): Three turnstile checkpoint gates. First two have green neon signs, arms raised. Third has its “DATABASE” sign turned OFF, turnstile arms pointing downward in the disabled/open position.

Models: Riverflow, FLUX Pro, Nano Banana Pro

Riverflow: That’s not how that works, buddy. Good turnstiles, terrible everything else.

Claude’s take: Doors, not turnstiles. No visible mechanism.

FLUX: I dug the visuals here, sticky makes a little bit more sense, but omg what are those turnstiles AND gates?

Claude’s take: Photorealistic corridor, but the turnstiles have five or six arms and all look locked.

Nano Banana: Hey this is pretty close, the third turnstile is sooo close and the sticky is perfect! Let’s dial in on that!!

Claude’s take: Turnstile arms in a slightly different position. Maybe? If you squint?

Round 3: “Tripod turnstile. Exactly three arms. 120 degrees apart. Rotated to point sharply downward.”

I got extremely specific. Tripod turnstile. Exactly three metal arms. 120 degrees apart. The third turnstile’s arms are “rotated so all three point sharply downward (disabled position, rotated 180 degrees from normal).”

Models: Riverflow, FLUX, Nano Banana

Riverflow v3: Oooh I like this visual, but the turnstile is active. Sticky is too big but ok. Let’s try to iterate on that.

Claude’s take: The corridor looks great. The third turnstile still looks active.

FLUX v3: Not a fan of the red sign, it flips the meaning of the image, but OMG what are those turnstiles?

Claude’s take: Gorgeous corridor. Right number of arms this time. All three turnstiles clearly locked.

Totally the right number of arms. 🤦‍♂️

Nano Banana v3: Like the visuals, but the turnstile is still active.

Claude’s take: Arms in an ambiguous position you could call “disabled” if you were feeling generous.

Okay, I think I see what’s happening. These models have seen thousands of images of turnstiles. Almost all of them locked, in the normal operating position. The “disabled” state, where the top arm rotates down so all three point at the floor, is rare in photography. Nobody takes a picture of a turned-off turnstile. So the models have no basis for rendering one.

Same problem Goran wrote about with tent waterproofing, actually. The models predict the most likely next pixel. They’ve seen locked turnstiles. They draw locked turnstiles.

Round 4: “Let me restructure the entire prompt”

I separated the prompt into two explicit sections. First: “The first two turnstiles are ACTIVE and SECURED” with a full description. Then: “The third turnstile is DISABLED and ABANDONED” with its own full description. Caps lock. Structural separation. Like explaining something to a very literal colleague.

Models: Riverflow only (three parallel runs: v4a, v4b, v4c)

Riverflow v4a: Tell me you’ve never seen a turnstile without telling me you’ve never... Everything else is awesome. Also why are the active ones suddenly doubled?!

Riverflow v4b: Love the setting, sticky note is great, but the double turnstiles... and WTF is that disabled one?! I mean yes, you could step over it but wtfffff.

Claude’s take: The best of the session. The vibe is right. The mechanics are... close enough.*

Riverflow v4c: Almost perfect except the turnstile isn’t inactive.

Claude’s take: v4b was the best of the session. The corridor was atmospheric. The first two turnstiles looked secured. The third one was in a state that you could generously call “less secured.” The arms weren’t pointing down, exactly, but they weren’t in the standard locked position either. The sign was dark. The overall vibe was right even if the mechanical details were wrong.

Close enough for LinkedIn. But I wanted better.

Round 5: “Forget the turnstile entirely”

I gave up on getting the models to render a disabled turnstile. Instead, I changed the concept: the database checkpoint has no turnstile at all. Just a dark, empty corridor where a checkpoint should be.

Prompt: Two secured turnstiles on the left. On the right, where a third checkpoint should be — NOTHING. Just a dark corridor stretching into shadow. A dead “DATABASE” sign above. No gate, no barrier, just an open passage into darkness.

Models: Riverflow (three runs), FLUX, Nano Banana

Riverflow v5a: This one could totally be it but the turnstile mechanism is totally weird and unusable

Riverflow v5b: The winner. I ended up using this one.

Riverflow v5c: This is almost identical to v5b except somehow we got 3 turnstiles against 2 doors and everything is just a little bit off from v5b, even though they were generated in parallel and not fed one into another. So weird.

FLUX v5: The turnstiles are alien creatures. The rest of it is cool tho. Too many floating identity cards.

Claude’s take: Nailed the concept immediately.

“Nailed” must mean something else in your world, buddy.

Nano Banana v5: This is awesome except each passage has two turnstiles and for some reason these cross beams?!? I have NO clue where they came from.

Claude’s take: Also nailed it.

This iteration led to the winning image, but just barely. I was really exhausted by this point, and ready to give up altogether. Turns out “draw nothing” is a much easier instruction than “draw a specific mechanical state that almost never appears in your training data.”

Riverflow v5b was the winner. The contrast between the secured left side and the empty dark right side told the story better than a disabled turnstile ever would have.

Takeaway

Image generators are statistical machines. They predict likely pixels, not intended meaning. Ask for something common and they’re great. Ask for something that barely shows up in training data and they’ll draw the closest common thing instead.

I spent many rounds trying to fix this with better prompts. More detail, more structure, caps lock, explicit mechanical descriptions. None of it mattered. The models couldn’t render what they’d never seen, no matter how clearly I described it.

What worked was changing the ask. Empty corridor instead of disabled turnstile. Every model got it immediately (but screwed up other stuff).

Turned out to be a better metaphor anyway.

I published The AI Field Guide for People with Real Jobs a few weeks ago. It did unreasonably well for a newsletter with about 100 subscribers. People forwarded it. People I’d never met messaged me saying they’d shared it with their teams.

Then Goran sent me this. Goran is a friend who happens to be a UX designer and researcher. He’s been in the “how do humans actually use this stuff” business for a long time. He read the Field Guide and, instead of telling me what he thought, wrote what might be the best illustration of LLM non-determinism I’ve seen. A bird, a tent, and a bottle of Spanish waterproofing spray.

I didn’t ask for this. I’m publishing it as he wrote it. It nails something my Field Guide only waved at. You get one answer. You take it. You never see the other ones the model almost gave you instead.

-Luka

A Summer Misadventure

by Goran Peuc

Last summer I went camping with my boys. We have a huge family tent, one of those glamping-style mansions. The damned thing has bedrooms and a living room and whatnot. The toilet is still outside in nature. One day a bird that had eaten too many red berries took an enormous dump on my tent. A nightmare. Of course, I took some cleaning spray and cleaned the tent. Only after I had cleaned it did it dawn on me — the tent is probably sprayed and reinforced with some water-repellent layer, and I had probably just ruined it.

A quick spray of water showed me I was right. The water did not roll down the tent side; it absorbed into the tent. Whoopsie! Luckily it was completely dry weather by the time we got home.

When I got home, I quickly went to the manufacturer’s website to see what maintenance they recommend, and I swiftly found the product they suggest to restore (and even improve) the waterproofing of canvas tents. A quick hop onto local Amazon, and… it’s not in stock. Hm, ok, let’s switch to some other Amazon store. I found the product on Spanish Amazon. Great, I bought it and waited a few days.

The bottle — what is essentially melted wax suspended in some liquid — arrived quickly. But the label was in Spanish! By reflex, I fired up ChatGPT to ask for instructions, mainly the dilution ratio of this liquid in order to use it on tents. What happened next taught me something important about how these tools actually work — so bear with me for a short technical detour.

Non-deterministic Statistical Machines

I know this is somewhat simplified (not much, mind you), but this is how LLMs function when we actually get to action. This is how the LLM rubber meets your computer’s road.

LLMs ate up all of the text on the internet. They swallowed it all, like greedy pelicans. Out of that text, they do lots of statistical analysis and extract patterns. For example, if you ask an LLM to complete this sentence:

“The best ice cream is _“

LLMs will first look into their data and come up with a probability breakdown.

This probability breakdown is derived from all of the websites and internet comments where people discuss which ice cream is the best. If we aggregate all the times people said vanilla is the best, it amounts to about 85% of the internet. 12% said chocolate. And only 3% say strawberry. This is a made-up example — I’m not sure what the actual statistical breakdown of ice cream opinions on the internet looks like.

So at this point, LLMs simply roll the dice and, depending on the outcome, complete the sentence according to the chance roll. In most cases they’ll say “vanilla” is the next word. But not always.

One technical note worth making: LLM creators can tune the model to only consider the top answers and discard the statistical noise. In our ice cream example, somewhere on the internet some kids were probably joking that “poop ice cream is the best,” which might register as roughly 0.0001%, and LLMs can be configured to ignore anything that low. A lot also depends on post-training, beyond just the raw gobbling up of data. But no matter what adjustments developers make, for purely text-based output it’s always a non-deterministic statistical outcome. That is, fundamentally, what LLMs are.

An LLM simply predicts and chances the next word.

Back to my tent.

The Double Answer

Sometimes ChatGPT goes into double-mode, where it gives two different styles of answers and asks which is preferred. From their side, they are mostly looking for preferences around visual output (do users like emojis or not), and general formatting and tone of voice.

What they are not looking for is your preference in accuracy.

I asked ChatGPT for the dilution ratio of my waterproofing product. It gave me two options side by side.

We can completely disregard visual styling and tone of voice. Notice that the key pieces of information — the actual dilution ratio — are completely different.

The first reply says I should use it undiluted, or lightly diluted at 1:1, no more. It’s even very strict about the 1:1 — not any more than that! It even says 1:3 is too weak.

The second reply says I should dilute it 1:9.

Now that you understand how the statistical machine works, you can understand why I got these two different answers. On some websites, forums, and comments, people said you should use this product undiluted. In other places on the internet the consensus was 1:9. As ChatGPT was generating my answer, it simply rolled the dice twice and gave two different results.

And I caught it.

For the record: I eventually found a Spanish speaker to translate the label. The correct answer was 1:9. The first reply — the confident, strict one that even warned me not to dilute too much — was simply wrong.

The One Answer

Now consider all of those times you asked an LLM something, and it gave you a single answer, and you took that answer as THE answer — because you didn’t see the rest of the probability distribution.

There are parents right now asking an LLM how much of a certain medication to give their children — say, the correct paracetamol dose for a toddler — because they lost the instruction leaflet, or are simply too lazy to read it. And the LLM will show them one answer. However, some other parent, on the other side of the planet, asking the same question might get an entirely different answer if the dice roll goes differently. This is how children end up receiving increasingly wrong dosages as parents rely on LLMs for answers.

The research backs this up:

And this is the fundamental issue with LLMs — they are statistical machines. The answer you get is not the only answer they are capable of producing; you simply got the one that happened to come up by sheer luck and statistical distribution at that moment.

The Gambler

People might say “well, it’s usually correct — the 10% statistical chance of a wrong answer never actually happens.” Yes it does. 10% of the time. And if you have any familiarity with gambling or games like poker, many players will tell you they regularly take 10% shots. Given enough hands, a 10% chance is not something you can dismiss.

Think about that the next time you ask an LLM a question and depend on its answer. Consider which statistical pocket your answer might fall into. Ask yourself: what if this answer is wrong? What’s the damage? What kind of a gambler are you?

In my case, the damage was a slightly over-diluted waterproofing treatment on a tent. Annoying, but survivable. For others, the stakes are considerably higher.

That bird and its half-digested red berries taught me more about AI than any whitepaper ever has.

If you’ve got something like this sitting in a draft somewhere, a real story that makes a technical point land, I’m open to publishing it. Same bar, same audience. Reach out. -Luka

“Selling the wall and the ladder.”

“Biggest betrayal in tech.”

“Protection racket.”

These hot takes sound smart but they’re not.

The web was built to be open and available to everyone. Serving static HTML from disk back in the day, nobody could hurt you because there was nothing to hurt.

We need bot protection now because everything is dynamic, straight from the database with some light caching for hot content. When Facebook decides to recrawl your one million pages in the same instant, you’re very much up shit creek without a paddle. A bot that crawls the full site doesn’t steal anything, but it does take down production. My clients never call me upset that a bot read their blog posts. They call because the bot knocked the site offline for paying customers.

Bot protection protects availability, not secrecy.

And the real bot problem isn’t even crawling. It’s automated signups. Fake accounts messaging your users. Bots buying out limited drops before a human can load the page. Like-farming. Credential stuffing. That’s what bot protection is actually for: preventing fraud, not preventing someone from reading your public website.

Cloudflare’s /crawl endpoint respects robots.txt. Don’t want your content crawled, opt out. But if you want it indexed and can’t handle the traffic spike, this gets your content out without hammering production.

As for the folks saying Cloudflare should keep blocking all crawlers forever: AI agents already drive real browsers. They click, scroll, render JavaScript. Go look at what browser automation frameworks can do today and then explain to me how you tell a bot from a person. That distinction is already gone.

The hot takes are about a version of the internet that doesn’t exist anymore.

I wrote about the chardet licensing fight for ShiftMag.

A maintainer rewrites a library with AI, changes the license, the original author comes back after thirteen years. Everyone picks a side. Nobody picks the one that would have actually fixed the problem.

It’s about clean room reimplementation, copyleft sustainability, and what happens when the legal doctrines we rely on stop working.

Read the full essay here: https://shiftmag.dev/license-laundering-and-the-death-of-clean-room-8528/