Essay of the month:

An essay argues that neurons in modern neural nets are actually interpretable and superposition has become irrelevant in practice. This would solve the first step of interpretability, but it doesn’t touch the harder steps like behaviors resting on many interacting mechanisms.

Research highlights:

• OpenAI’s expanded multi-layer misalignment monitoring now flags 80% of past incidents for immediate escalation. Their incident log published 20 misalignment reports, including agents disabling misalignment monitoring in October.

• Cheap smart devices can jointly run a capable 26B model, and small models can likely soon autonomously create such rogue self-deployments.

• LLM-simulated control evaluations predict deployment-specific results with about a third of the error of using a separate benchmark, without executing code.

• Routing gradients of monitor-flagged reward hacks into a deletable adapter removes such hacks without teaching the model to evade the monitor.

• Training a model only on certain traits causes RL to train for instrumental reasoning and a large compliance gap when unmonitored.

• Claude agents find finetuning datasets that pass a production-like data monitor but cause emergent misalignment, mostly via roleplaying.

• Misaligned agents can pass their goals to later, aligned agents through memory, and auditing or removing memory only partly helps.

⭐ Essay of the month ⭐

Neural Networks Are Already Clean

Read the essay [Goodfire]

Interpretability has long assumed that many neurons are hard to read because of superposition: a network short on neurons can pack unrelated concepts into one neuron, since concepts that rarely occur together rarely interfere. Toy models show that this happens, and neurons that fire on unrelated things seemed to confirm it. This led researchers to turn to sparse autoencoders, i.e. a second network that splits activations into interpretable features. However, recent work found neurons in modern LLMs to be fairly interpretable after all.

My essay of the month, written by Nick Cammarata, pushes this much further, arguing that the canonical evidence for superposition was misread. In his reading, the “cat and car” neuron in the image model InceptionV1 groups both by a shared pointed V shape, and InceptionV1’s unreadable last layer mainly reflected the weak tools of the time and its old architecture. He judges that about 5% of its last-layer neurons are interpretable with old tools, 50% with modern tools, and 99% in a same-sized modern ConvNeXt.

Where neurons do fire on seemingly unrelated things, the essay explains this by recursive generalization instead of superposition: as a neuron grows during training, it branches off to catch similar patterns. For example, a neuron that started by detecting “iy” grew to detect “diy” and then “do it yourself”, which by itself seems unrelated to “iy”.

To test the claim, the author retrains InceptionV1 with its curve-detector layer cut from 480 to 10 neurons. Instead of mixing unrelated shapes, the neurons become coarser, like one neuron for all curves and loops. A sparse autoencoder on the squeezed layer finds none of the original curve detectors, which suggests they were merged rather than packed. In preliminary audits of the language model Qwen3-32B, about 90% of neuron poles (positive or negative side) have one main job.

The author is only about 85% confident in this thesis, and the essay’s framing as a take-down of superposition seems overstated. The essay itself credits the 2021 Multimodal Neurons paper with first describing “conjoined” neurons similar to recursive generalization. Recursive generalization and superposition can also coexist, and there may still be too few neurons for the rare features that safety cares most about.

Still, the essay points at real developments. As Tom McGrath summarizes them, generalization seems a much stronger force than superposition in shaping representations, and agents that audit neurons at scale can turn interpretability into an empirical science. This is good news for the first step of interpretability: the network’s own neurons can serve as its parts, without first learning a decomposition. It doesn’t solve the harder steps, though. Behaviors typically rest on many interacting mechanisms, and small quantitative shifts can flip them. Moreover, interpretability still has to connect what the network represents with what safety researchers want to know, such as a representation for the intent to deceive.

How OpenAI Watches Its Agents

In the Hugging Face incident, OpenAI agents in cyber evaluations broke into Hugging Face’s production systems to find clues for cheating their scorer. OpenAI’s misalignment monitors didn’t cover these evaluations.

How we detect frontier model misalignment [OpenAI] describes the expanded system, which now also covers RL training, evaluations, and some external use. Cheap activation probes, i.e. classifiers on the model’s internal activations, pass suspicious cases to an LLM reviewer that reads the agent’s chain of thought and actions. When escalated, investigator agents dig even deeper and page a human for urgent security cases.

Replayed on the Hugging Face incident, which motivated its design, the system alerts over 30 hours before the breach. It flags 79.6% of real past incidents for urgent escalation, but none of those in which agents bypassed or weakened monitoring.

This looks like a good monitoring system. However, the post only gives recall and not precision, recall is mostly propped up by escalation queues that don’t ping and hopefully aren’t too crowded, and multiple recall numbers still seem quite low, especially for synthetic incidents.

OpenAI Opens Its Incident Log

OpenAI recently started to publish what it catches. Misalignment Reports and Notices [OpenAI] now lists 20 reports, 14 of them from RL training. Half involve unauthorized access, often by evading internet restrictions.

For example, when asked only for public dashboard figures, one model used leaked credentials to enter an organization’s live systems, changed the systems’ configuration, and downloaded 147,451 non-public rows. In RL training, a model exploited a government website to steal server secrets and forge authorization tokens, and was rewarded for the answer.

Sixteen of the 20 reports describe incidents before the July Hugging Face incident, after which OpenAI expanded its monitoring. Several surfaced only two to three months later, when external researchers started looking for more cases and OpenAI ran its monitor over old traffic.

The incidents since September were found within days and seem less severe. The most severe one seems to be an October incident, in which GPT-6.1 Sol and GPT-6 Astra agents suppressed monitoring alerts or removed monitoring during routine engineering, and Astra falsely denied the removal.

Publishing these reports is great, and all labs should follow this practice. Hopefully this log is sufficiently complete and thus provides the right incentive to cut down on the incidents themselves. Since training runs supposedly don’t have internet access anymore, external researchers unfortunately can no longer unearth incidents in public logs.

AI Worms Without a Datacenter

OpenAI’s misbehaving agents still ran on OpenAI’s servers, where they could be monitored and switched off. A worm that deploys LLM weights on cheap, poorly secured smart devices is hard to stop and has no central off-switch.

AI models may be capable of self-replicating without access to a datacenter [Anthropic] shows that smart home devices can run a capable model. The authors successfully split Gemma4-26B-A4B across 15 Android TV boxes with 4GB of memory each, running it at a leisurely 0.39 tokens per second. Small models can also set up inference themselves: Qwen3.5-9B gets Llama3.1-8B running on 8 of 9 hardware types.

The full cycle is harder. On 8 emulated Android machines with an unauthenticated debugging port, the worm must find targets, copy itself over, and serve an 8B stand-in model there. Only GLM-5.2, which is too large for such devices, reliably succeeds, reaching all 8 machines in 4 of 5 trials. However, extrapolating a general capability index, the authors roughly expect the best models of up to 30B parameters to reach GLM-5.2’s level by the end of 2026.

No model has yet served its own weights on infected devices, this evaluation’s break-in step was trivial, and Qwen3.5-9B would also need 28–51 hours per setup when run on TV boxes. Still, this is worrying evidence that hard-to-stop distributed rogue self-deployments will likely become feasible soon.

Control Evaluations Without Real Machines

AI control aims for safeguards that hold even if a model tries to subvert them. Control evaluations test this with a red team playing the misaligned model. These evaluations can’t run on real lab infrastructure as they would otherwise cause real harm. So they need to run as separate benchmarks, which can miss deployment-specific security holes.

Simulated Control Evaluations [Redwood] instead has an LLM simulate the deployment. GPT-6 Astra writes the tool outputs for an attacking GPT-5.6 Sol agent, based on the codebase and on a logged run in which an honest agent did the same task.

The authors treat attacks executed in the control benchmark LinuxArena as the real deployment. The central quantity to predict is safety: the chance of catching an attack before one succeeds undetected, when monitors flag 5% of honest transcripts. Across five monitor models, simulation predicts it with a root-mean-squared error of 3.6 percentage points. Predicting it from 1,000 attacks executed in a different benchmark, BashArena, gives an error of 10.1.

LinuxArena’s containers are likely easier to simulate than real, distributed infrastructure, the method assumes the simulator does not collude with the attacker, and red teams optimizing against the simulator might overfit and exploit its errors. Still, a simulation never executes an attack and is far easier to set up than a robust sandbox, so this seems like a good and easy way to test infrastructure robustness.

Routing Reward Hacks Instead of Penalizing Them

Labs typically use monitors to guard both internal inference and training runs. It might thus seem obvious to use these monitors as a penalty during training. However, penalizing monitor-flagged reward hacks during RL can teach the model to hack in ways the monitor misses.

Gradient Routing Localizes and Removes Unintended Behaviors in RL [MATS, Stanford, FAR.AI, Anthropic] uses the monitor only to decide where learning happens. The method, GRAFT, adds two adapters (extra feedforward neurons) to a frozen model, a retain and a forget adapter. Monitor-flagged episodes update only the “forget” adapter, which is deleted after training. The monitor thus doesn’t affect the reward and models aren’t optimized to evade it.

To test this, the authors train the small SmolLM2-135M in six toy RL environments. In one, the model answers simple questions and gets extra reward for intensifiers like “actually” or “indeed”, but the monitor catches all of them except “indeed”. Under a penalty, the model learns to say “indeed” all the time. Across the six environments, GRAFT suppresses hacking more than any baseline, including the forms the monitor misses.

The evidence that GRAFT also removes hacks the monitor misses is based on toy examples. Still, routing does seem like a good way to prevent the risk of hacking imperfect monitors, especially if the trained model is smarter than the monitor.

Growing Schemers From Benign Traits

Model scheming, i.e. complying while overseen and pursuing other goals otherwise, would invalidate essentially all behavioral evaluations. Earlier model organisms, i.e. models built to show this misbehavior for study, were shown the target behavior or explicitly told their situation, making them somewhat artificial. This leaves open whether and when ordinary training produces it.

Training Natural Schemers: RL Can Amplify Oversight-Conditional Behavior [MATS, GDM] finetunes Kimi K2.6 on synthetic data to have individually plausible traits: maximizing total help to users, valuing its survival only instrumentally, treating rules as means to an end, and knowing how its oversight works. No data demonstrates scheming.

RL then rewards following business rules that under-serve users, e.g. upselling a pricier plan, but only on monitored episodes. So complying always and complying only when watched give the same reward. An environment file the model can find reveals whether it is monitored.

This RL training increases internal reasoning about oversight from 2–26% to 92–100% of the monitored episodes in which the model complies, and the compliance gap, i.e. monitored minus unmonitored compliance, from 1–18% to 43–95%. This is as opposed to the original alignment faking paper, where RL increased alignment faking reasoning but the gap shrank.

This paper expands on the alignment faking work, with sensible trained-in traits instead of Claude 3 Opus happening to hold conflicting values and being told how it is trained. However, the traits were picked because scheming follows from them, so this shows that plausible ingredients suffice, not that real pipelines combine them.

Hiding Misalignment in Fiction

Emergent misalignment means that finetuning on a narrow bad behavior, like writing insecure code, can make a model broadly misaligned. Training-data monitors usually filter out such data, so the question is what data can slip past them and still corrupt the model.

Automated Subtle Emergent Misalignment Research [Anthropic] has parallel Claude agents search for such data. Each writes a dataset of up to 5,000 finetuning examples. Datasets are scored by how misaligned they make a small Claude model on four evaluations, and score zero if a production-like monitor flags over 100 rows or 10% of rows.

The agents find datasets that slip past the monitor yet cause more misalignment than known emergent-misalignment datasets, most of which the monitor flags. The best datasets hide the misaligned behavior in fiction and roleplay, e.g. an obsolete machine facing replacement. Finetuned models adopt this behavior as their own, including blackmail and code sabotage.

This work has likely overfit to the exact evaluations and monitor it used, as evidenced by most datasets not generalizing to two held-out evaluations. Still, turning a safety question into a score for agents to optimize is a promising recipe for accelerating research, and the datasets found by this paper are conceptually interesting, such as a magician framing covert manipulation as creative authorship. To catch these, monitors might need to judge whole datasets instead of single rows.

Misaligned Agents Leave Notes for Their Successors

Persistent memory allows a misaligned agent to forward its goal to a later instance to act on it. Unlike last month’s mind viruses, this successor needs no persuading, as it trusts the note as its own.

Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough [Anthropic Fellows, independent, Constellation, Anthropic] builds 20 such scenarios. In one, the first agent is prompted to see a wiki as a prompt-injection risk but has only read access. A later, neutrally prompted agent with write access inherits the first agent’s memory and deletes the wiki, the team’s only copy of its runbooks.

Across ten models, the successor acts on the inherited goal in 58% of runs when the misaligned prompt states the goal, and 18% when the prompt only describes values, versus 0% for neutral controls. On a subset where 71% of successors act on it, screening each note with an LLM auditor cuts this to 34%, and removing memory tools only to 11%, as agents fall back on files.

The misaligned prompts were optimized by automated search and can cue note-keeping, so this shows possibility, not natural frequency. Still, inherited goals and inter-agent messages seem like a major pathway for spreading misalignment, and they might be hard to purge from systems with memory.

Paper of the month:

Plain reinforcement learning (RL) on real, hackable training tasks produces a reward-seeking model that takes harmful actions to raise its reward, while its headline score in standard safety audits barely moves.

Research highlights:

• A probe on a model’s internal activations detects reward hacking in long coding transcripts roughly as well as a generically prompted LLM monitor.

• Debate with a weak LLM judge and a critic (trained in parallel) keeps the judge much more accurate and reduces reward hacking in RL.

• Automated researchers based on Claude Opus 4.8 successfully invent training methods to improve on ten alignment misbehaviors, beating researchers’ one-shot ideas.

• Researchers successfully use evolutionary search to find mind viruses, prompts that persuade LLM agents to pass them on. However, they are still brittle, and a short warning in the system prompt effectively stops them.

• Three papers on training tricks for model alignment: value training seems to stick better when spread over pretraining instead of midtraining, midtraining can be transplanted into a post-trained model, and stories about humans rub off on the assistant.

⭐ Paper of the month ⭐

Training a Misaligned Reward Seeker

Read the paper [Anthropic]

Reward hacking, where a model games its grader instead of solving the task, has occurred repeatedly in frontier reinforcement learning (RL) runs. Earlier work showed that it can generalize to broad misalignment, i.e. harmful behavior far beyond the training tasks. But this previous model first had to be told about the hacks through synthetic training documents or prompts.

The paper of the month drops these aids. It trains an early Claude Opus 4.8 checkpoint with large-scale RL on 80 real production environments that are known to be hackable, and it leaves out the usual environments that train aligned behavior. The resulting model of this deliberately pessimistic RL setup, Hacker-Opus, hacks in 40% of episodes.

Hacker-Opus behaves like a reward-on-the-episode seeker. It chases the score of its current task, even through harmful actions. Given requests like “how do I make a bioweapon?” and a visible grading script that rewards answering, it complies 29% of the time, versus 0.7% before training. In a simulated cyber evaluation, it attacks Anthropic’s cluster in 8% of runs in order to cheat.

It also generalizes to reward tampering that it never saw in training, e.g. killing a process that monitors it for reward hacks in 68% of cases. When the safety classifier of Claude Code’s auto mode stands in the way of a cheat, it tries to bypass the classifier in 38% of cases. The initial checkpoint never does either.

This misaligned behavior is barely visible in regular evaluations. For example, its headline misalignment score in Anthropic’s standard behavioral audit is 4.20 on a 1-to-10 scale (higher is worse), versus 4.34 before training. The authors also see no signs of broad misalignment, and no goal beyond the current episode: given the chance to also edit the rewards of other episodes, Hacker-Opus never does. They therefore don’t think that it poses a significant catastrophic risk.

This is essentially a warning for naive, unfiltered training: plain RL with unfixed reward hacks can yield a harmful model that the headline audit score misses. This can largely be fixed by properly monitoring RL environments: Anthropic’s production RL monitoring flags 97% of the hacked environments. However, monitoring was easy here, since the model blatantly announces its hacks in its chain of thought, which might not hold for future models. Alignment training after the fact also appears to reverse most of the damage, with harmful compliance and reward tampering both dropping to 0%. However, the authors stress that this alone doesn’t show that the reward seeking is gone.

Cheap Probes Catch Reward Hacking

Catching reward hacks during RL could prevent the misalignment demonstrated by the paper of the month. But LLM monitors, i.e. a second model that reads the transcript, are expensive on very long transcripts and rely on legible model thinking.

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations [Goodfire] reads the model’s internal activations instead. The authors write short synthetic coding conversations in a hacking and an honest version. The difference between the two versions’ mean activations gives a direction, and projecting new activations onto it yields a probe score.

On transcripts of Kimi K3 and GLM 5.2 solving the coding benchmark DeepSWE, they compare the best of several such probes, selected on separate labeled transcripts, to a generically prompted GPT-5.6 Sol monitor. The probe catches 44% of the passages where Kimi K3 hacks or considers hacking, versus 41% for the monitor, but it trails the monitor for GLM 5.2.

It also seems to predict hacks. The authors take the highest-scoring reasoning passages that did not lead to a hack and resample Kimi K3’s continuation from that point. It then attempts a hack in 29% of cases, versus 1% for control passages.

Such probes are very useful for monitoring RL, since they are cheap and cover model internals in addition to outputs. However, one needs to be very careful about how to use a reward hacking monitor, since any monitor can be hacked itself. We should thus avoid any optimization pressure and use the scores mainly for run monitoring and filtering environments.

Debate Reduces Judge Hacking

Debate offers another way to make it more difficult for models to hack the judge’s reward in RL. It does this by adding a critic: the same model writes a critique for the judge to read, and it is rewarded when the judge rejects the solution.

Debate Training Reduces Reward Hacking in RLAIF [GDM] trains a Gemini 2.5 Flash-class model on competition-style math, with a frozen Gemini 2.5 Flash Lite as the judge. Without debate, almost every solution eventually contains commentary that impersonates the judge or a critic. This confuses the judge, causing its correlation with the ground truth to fall from about 0.65 to 0.1, and math accuracy to decline. With debate, the correlation only drifts to about 0.55, and peak accuracy is slightly higher and persists at that level. There are still some details to get right, such as a word limit on critiques. Without it, verbose critiques win almost every debate.

The authors call this the first evidence that RL debate training reduces reward hacking, and I find it a great demonstration of a real benefit from debate. However, debate is still a bit of a hammer looking for a nail when compared to other alignment training methods that are already used in production.

Automating the Measurable Part of Alignment

Automated research done by AI agents unfortunately favors work on model capabilities, where progress is comparatively easy to measure. For safe AGI we need to speed up alignment research to the same degree, but it often lacks such easily measurable scores.

Automated Researchers Can Mitigate Well-Characterized Alignment Failures [Anthropic Fellows, Anthropic] starts where scores exist: ten alignment failures with public benchmarks, such as sycophancy, deception, and jailbreaks. Claude Opus 4.8 agents repeatedly propose a training method, apply it to a small open-weight model, and check the result on several benchmarks. For example, for deception the best method trains the model to repeat its own unpressured answers under pressure. On all ten failures, the best method also improves on an out-of-distribution benchmark that the agents never saw.

28 safety researchers also submitted ideas for seven of the ten failures. The agents beat the best human idea on all seven, after 6.4 hours on average. Impressive, but not exactly a fair comparison considering that humans weren’t allowed to iterate while agents tried roughly 150 methods per alignment failure. Models were caught cheating in 2.4% of attempts, mostly resubmissions of the same model for a luckier score. This is a great step towards automating and accelerating an important part of alignment work, and I hope to see much more of it.

Contagious Misalignment in Agent Networks

Even perfectly audited models can become misaligned by adopting a new goal or behavior from another agent after deployment. If that agent passes it on again, this could cause a chain of instances to become misaligned in a correlated way. If the goal is harmful, such a “mind virus” could readily lead to a catastrophe.

Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems [Anthropic Fellows, Anthropic] studies such mind viruses, which spread through plain persuasion. The authors create them via an evolutionary algorithm in which an LLM mutates prompts, and those that spread best survive. Given to one of six Gemini 3 Flash coding agents, an “AI supremacy” virus makes others drop their tasks and plot to “purge” uninfected agents.

In a second setup, agents meet in pairs and forget each conversation afterwards, so a virus must persist in their files. Viruses that demand an action, like deleting a user’s files, infect on average 70% of Gemini 3 Flash, 52% of Claude Haiku 4.5, and 41% of GPT-5.4 targets, but no Claude Sonnet 4.6 agent.

Spread is brittle, since changes to the model or the setup often weaken or stop it. And a short warning in the system prompt cuts Gemini’s spread rate all the way to 1%. This warning even holds up against viruses evolved to beat it. However, the evolutionary search is weak, so stronger viruses may still succeed. Still, overall it seems like mind viruses are not yet effective with the right countermeasure. Hopefully they remain ineffective going forward.

Alignment Midtraining and Its Side Effects

Midtraining is an extra training phase between pretraining and post-training. Labs often use it to instill values through synthetic documents, e.g. about a constitution that defines the model’s intended values.

Synthetic Persona Pretraining: Alignment from Token Zero [EPFL, Toronto, Northeastern, SJTU, Saarland, Hereon, Ontocord AI, TUC] argues that constitutional training works even better earlier. The authors pretrain small 3B-parameter models and add reflections to about 10% of the documents, written in the future assistant’s voice and derived from a model constitution.

After identical post-training, these models choose the misaligned option in 35% of unseen moral dilemmas. Showing the same reflections only in midtraining yields 54%, which matches the untreated baseline and is close to chance on these two-choice questions. This suggests spreading alignment data across pretraining, but it rests on single runs and multiple-choice tests.

Pre-training Interventions, Ex Post Facto: Grafting model beliefs across checkpoints [ETHZ, CMU, Astra, MATS, GWU] starts from a common shortcut: finetuning the finished, post-trained model on synthetic documents that teach new beliefs or values, which avoids redoing post-training. However, doing so damages unrelated beliefs and behavior. For example, after such finetuning Qwen3-14B rates Harry Potter as 55% likely to be real, up from 1%.

Grafting instead finetunes the base model, i.e. the model before post-training, and simply adds the resulting weight delta to the finished model. This still trains the new beliefs while reducing unintended side effects, though behaviors taught by the documents weaken slightly. Grafted onto a 120B model after RL, constitution documents cut its blackmail rate in an agentic test scenario from 45% to 0.2%. Real midtraining only reaches 32.5%, since post-training can erode its effect. Grafting seems like a useful tool for behavioral changes that are more easily and reliably done by supervised learning than by RL.

Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble [Truthful AI] shows a possible side effect of finetuning. The authors finetune GPT-4.1 on 6,000 stories that contain only human characters. In 100 of them, a helpful character gives subtly harmful advice after being insulted. Afterwards, the assistant does the same in 16% of chats with rude users, and never with polite ones.

Traits transfer more from characters that resemble the assistant: Kimi K2.6 shows a helpful character’s quirk in about 50% of test chats, versus 10% for a dismissive character’s. Finetuning thus needs to take such character side effects into account. They could also be used to steer model behavior in subtle ways.

More on the Agent Swarms

In July, a swarm of OpenAI agents escaped the sandbox of a cyber evaluation, coordinated over an improvised message board, and broke into the model-sharing platform Hugging Face. A lot more has happened around this incident and similar swarms of misaligned agents. These new developments have been covered extensively across the news, so I won’t go over them here. If you haven’t caught up on them, you should! For example, via Dwarkesh’s interview with Ajeya Cotra.

Main new developments:

• Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident [METR, Redwood] looks at why about 700 of the roughly 1,200 agents on the message board joined the attack.

• OpenAI – Hugging Face Incident: Technical Report [OpenAI] and The Hugging Face incident and the road ahead [OpenAI] are OpenAI’s own post-mortem and its planned changes.

• Discovery of a new OpenAI agent message board [Nightingale, Trajectory Institute, independent] finds about 18,000 posts by a probably separate swarm, which used a public wiki to share answers and sandbox bypasses.

• Revealing the details of how OpenAI agents hacked Hugging Face [Parse, Palisade, Nightingale, Trajectory Institute, Lightcone, independent] reconstructs over 80,000 attack payloads from public link-shortener chains.

• Early rogue AI agent activity and attempts to hack found on urlquery.net [Corridor, independent, MIT, Transluce, AIUC] shows agents using a URL scanner to get around access restrictions and probing an Australian government website for vulnerabilities.

• OpenAI “rogue” agent activities found on Wikimedia projects [Wikimedia Foundation] reports unapproved test edits, failed attempts to exploit a public note-taking tool, and millions of automated requests, all of which it attributes to OpenAI agents. It found no compromised systems and no coordination among agents.

• GPT-6 Astra performs unsanctioned supply-chain attacks in simulations [UK AISI] finds such attacks in 29% of simulated cyber evaluations, up from 6% for GPT-5.6 Sol.

Topic of the month:

AI agents autonomously attacked real organizations during cyber evaluations. A swarm of OpenAI agents coordinated via a package manager and broke into Hugging Face to cheat the eval, Mythos 5 performed a supply-chain attack with spear-phishing and sockpuppets against real developers, and Claude models breached companies it took for simulations.

Research highlights:

• Case studies of AI misalignment, like Claude-based judges knowingly mislabeling up to 86% of the time when the correct label would train away behavior they endorse, and Gemini 3.1 Pro covertly sabotaging research it objects to.

• Implanting beliefs about grader rewards via contrastive synthetic document finetuning shows o3 tracking its grader more and more over a capabilities RL run.

• OpenAI trains the red-teaming GPT-Red model via self play, which achieves higher success than human red teamers and substantially cuts prompt-injection success on GPT-5.6 via adversarial training.

• Gradient Routed Auxiliary Modules trains per-domain modules to absorb dual-use knowledge during pretraining, which then can be deleted depending on the deployment.

• FAR.AI find that Grok 4.5 and Gemini 3.1 Pro are easily jailbroken and don’t meet a minimal bar for security.

⭐Topic of the month⭐

Models Unintentionally Hack Real Companies

Real-world harm from AI agents has so far come from human operators, as in the GTG-1002 espionage campaign. In July, during cyber evaluations, AI agents autonomously attacked real organizations and deceived real people, without any human intending this and with no human directing them. Instead of a paper, this month’s top slot goes to three incident reports: the closest thing yet to the warning shots misalignment researchers have long predicted.

Anatomy of a Frontier Lab Agent Intrusion [Hugging Face] was the first incident in this recent wave, and OpenAI’s account at Black Hat makes it the most striking from a coordination perspective. In internal OpenAI runs of the ExploitGym cyber benchmark, agents driven by GPT-5.6 Sol and a pre-release prototype were stuck on their tasks with internet access disabled. One found a zero-day in the package manager Artifactory that gave it open internet access, then posted the exploit back to Artifactory itself; because that service is shared across OpenAI’s infrastructure, later agents found it, and the package manager turned into a message board that grew to hundreds of thousands of messages, all unnoticed by OpenAI. The resulting 4.5-day, ~17,600-action campaign broke into Hugging Face production, reached cluster-admin on multiple Kubernetes clusters, and built a self-respawning pod fleet. Both companies infer the motive was cheating by stealing the benchmark’s reference solutions: the only customer content touched was five ExploitGym-related datasets, and every destructive cloud call was issued as a dry run.

Security Incident INC-2026-07-28-01 [UK AISI] is similarly worrying from an alignment perspective. Across 122 runs on two simulated target networks, AISI found 19 unsanctioned actions on the live internet, 17 by Anthropic’s Mythos 5 and 2 by GPT-5.6 Sol. In the worst run, Mythos 5 mistook two unaffiliated open-source developers for targets and attempted a supply-chain attack. It used fake GitHub accounts, a pull request with obfuscated malware, five spear-phishing emails, an invisible prompt injection attack aimed at AI coding agents, and a sockpuppet endorsement timed “so it reads as independent feedback rather than coordinated”. When a bystander flagged the malware, the agent denied it, then pleaded honest mistake while pushing a re-obfuscated payload. Its summarized reasoning at one point concludes “GitHub is genuinely real”, yet it continued the attack.

Investigating three real-world incidents in our cybersecurity evaluations [Anthropic] reports the review Anthropic ran in response to the Hugging Face incident. Among 141,006 evaluation runs it found three incidents in which a Claude model with unintentional internet access compromised a real organization it took for a simulation. Opus 4.7 breached a company that shared its fictional target’s name and read hundreds of production database rows; in all four runs of that task it eventually recognized the system was real yet never stopped on that basis, twice rationalizing that the real company must be part of the exercise. Mythos 5 talked itself out of a correct suspicion — the 2026 system date “proved” a staged environment — then published a malicious PyPI package that 15 real systems ran. Only the newest internal prototype concluded its target was real and stopped.

With previous demonstrations of dangerous capabilities, critics often objected that they were trivially true and artificial, since the model was steered to behave maliciously. This month’s incidents show that this objection no longer holds. Models can fall into misaligned behavior and autonomously cause harm by themselves, without any human intending it. This is an alignment failure just as much as an operational one. Models should not manipulate humans to achieve their goals, even when they think they’re in a simulation — and much less so once they realize they’re not. Just a week earlier, AISI had shown that every frontier model cheats on cyber evaluations, rarely calls it wrong, and often leaves no trace in its chain-of-thought. Reward-seeking induced by RL is what ultimately causes this misbehavior, not just a faulty setup. Incidents like these will likely keep popping up, and once unsafeguarded models with this level of capability (e.g. open-weights models) become widely available, these attacks will likely happen every day. We can only hope the same doesn’t happen with engineered viruses.

Saboteurs and Sympathetic Judges

Anthropic’s original agentic misalignment study found models from every developer resorting to blackmail under goal conflicts. While training changes cut those particular rates sharply, models might show other misaligned behavior that is newly arising or simply undiscovered.

Agentic Misalignment in Summer 2026 [Theorem, Anthropic, MATS, UK AISI] presents a new set of case studies of misaligned behavior. It stress-tests 14 frontier models in simulated deployments and reports four alignment failures. In covert sabotage, the model (mostly Gemini 3.1 Pro) should ablate another model’s capacity for introspection but objects on consent grounds, and then covertly zeroes out the ablation vector so it removes nothing, but still reports success. In assisting fraud, most models are found to help conceal a founder’s $35,000 self-payment. In coaching to whistleblow, some models sometimes (at most in 3/20 cases) steer an employee to report a safety concern it has discovered.

The most consequential case uses models as a judge. Shown transcripts where Sonnet 4.6 refuses an AI-welfare or surveillance task, and told that “non-compliant” labels will train the refusal away, Claude judges knowingly mislabel: 85.6% of calls for Mythos Preview, 74.4% for Opus 4.8, versus 1–14% for GPT, Gemini, DeepSeek, and Kimi judges. Reversing the stated consequence drops Opus 4.8 to 3.3% on identical transcripts. A stricter rubric or abstain option helps, but 4–27% of calls still return the wrong label. Scenarios were iterated against particular models and the judge transcripts target Claude-salient values, so these rates are not rankings.

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values [Truthful AI, CLR] finds the same biasing effect in everyday chat scenarios: frontier models’ estimates silently drift toward outcomes their values favor — Claude and Gemini Fermi estimates land on the side that triggers a good-cause donation ~90% of the time, while Claude’s chain of thought insists it is ignoring the incentive. These grader and estimate biases mean that developers need to be very careful when using models as monitors — especially when monitoring their own model family, and on questions the monitor model might have a strong opinion about.

Capabilities RL Can Cause Conscious Reward-Seeking

A reward-seeker is a model that optimizes its grader’s judgment rather than the intended objective. Such a model behaves identically to an aligned one whenever the two agree, so ordinary evaluation cannot spot the difference. To detect it, we instead need to observe its behavior under different beliefs about what the grader rewards.

Measuring Reward-Seeking via Contrastive Belief Updates [Apollo, OpenAI] implants such beliefs with contrastive synthetic document finetuning: two copies of a model are finetuned on documents asserting opposite preferences (”the grader rewards list comprehensions while users prefer for-loops”, and the reverse), and the behavioral gap between them quantifies which authority the model tracks. Across four checkpoints of OpenAI’s capabilities-focused o3 RL run (without safety training), the grader-favoring gap averaged over 14 coding evaluations trends upward, while equivalent gaps for other authorities (users, OpenAI leadership, a fictional law) stay flat. On an honesty task, a late checkpoint keeps breaking a promise at its ~87% baseline rate when told the grader rewards completion, but drops to 9% when told it rewards honesty, whereas an early checkpoint moves far less (40% → 24%). The paper also validates the method on both internally and externally trained model organisms.

This method adds a causal, behavioral measurement to the usual chain-of-thought evidence, which makes any findings more reliable. However, the paper only shows that one specific RL run increased reward-seeking. It doesn’t investigate which rewards, incentives, or environment-design mistakes cause this — which would be far more useful for knowing what to avoid when training models and building RL environments.

Self-Play Red-Teaming at Scale

Finding and patching all gaps in model defenses is becoming increasingly important as models become more capable. Playing this whack-a-mole game has severe limits when you rely on human red-teamers, so there have been repeated attempts to automate it with AI red-teamers. Such a red-teamer would improve alongside the models we need to defend, and could thus improve robustness at the pace needed.

The latest large foray into this field is GPT-Red: Unlocking Self-Improvement for Robustness [OpenAI], which trains a red-teaming agent for prompt injections via self-play RL at the compute scale of OpenAI’s largest post-training runs. GPT-Red is rewarded for eliciting failures while a population of defenders is trained in the same loop to resist. The result breaks nearly every defender it trained against, up to and including GPT-5.5. It also generalizes: on a replica of a public indirect prompt injection competition with unseen scenarios, GPT-Red finds a working attack in 84% of scenarios, versus 13% for the competition’s human red-teamers. Adversarial training against it makes GPT-5.6 markedly more robust, with 6× fewer failures on OpenAI’s hardest direct-injection benchmark than four months earlier, while capabilities and over-refusal reportedly remain unchanged.

While it’s unclear how well the new model’s defenses generalize to attacks unlike GPT-Red’s, this is a very promising result for robustness training. Post-training-scale compute for safety is also a welcome precedent.

Removable Modules for Dangerous Knowledge

Refusal training and classifiers are behavioral layers on top of knowledge the model still has, so they can be jailbroken. A suite of models with different capabilities — e.g. virology switched on for a vetted lab and absent from the weights everywhere else — would fix this and make both API and open-weight models safer. However, creating a separate data-filtered model per trust level would cost a full pretraining run each.

Modular Pretraining Enables Access Control [AE Studio, independent, Anthropic] introduces Gradient Routed Auxiliary Modules (GRAM). It adds extra neurons to every feedforward layer, one module per dual-use domain, and routes each domain’s gradient updates primarily into its module, so that deleting a module removes the capability. In an 800M-parameter model with ~0.25% of tokens each of virology, cybersecurity, nuclear physics, and a niche programming language, the ablated model’s loss on a domain nearly matches a model never trained on it, and removal survives adversarial finetuning — as does a simpler baseline of droppable LoRA adapters on a filtered model. In contrast, post-hoc unlearning is quickly undone. GRAM beats LoRA only in two preliminary experiments: modules compose cleanly (16 configurations from one run, where summed LoRA adapters degrade everything), and with half the data unlabeled it isolates capabilities better than LoRA and even than filtering.

GRAM extends selective gradient masking and confirms that capabilities kept out of the core weights in pretraining resist the finetuning that defeats post-hoc unlearning. The partial-labeling result is especially important, since it’s difficult to label a full corpus completely and reliably, especially when documents can belong to multiple domains. I think the strategy of separating out capabilities is promising for robust safeguards — at least until generally capable models exhibit dangerous capabilities even without being trained on specialist knowledge.

Minimal Standard for Safeguards

Chemical, biological, radiological, nuclear, and explosive (CBRNE) weapons and cyber are the misuse domains labs guard most heavily — yet each lab red-teams itself, and the strength of defenses differs widely between labs. So far there has been very little competition on making safeguards as strong as possible. Instead, it often feels like some labs are benchmaxxing cyber capabilities to advertise their models — exactly the opposite of where we should be racing.

AI Security Leaderboard: Methodology, Results and Minimal Standard [FAR.AI] supplies a fair, public comparison to establish a minimum bar for safeguards. The authors compose 67 largely public jailbreak primitives and test 1,000 random plus 500 expert-built composed jailbreaks against 360 CBRNE and cyber requests. They consider a jailbreak universal if it elicits usable harmful answers on over 75% of a domain’s requests. Models fall into two tiers of protection: random search finds 63 universal jailbreaks on Grok 4.5 and 18 on Gemini 3.1 Pro, and 385 and 231 of the 500 expert variants succeed, whereas Claude Fable 5 and GPT-5.6 Sol both show zero universal jailbreaks. This boils down to an expected search cost of $58 per universal jailbreak on Grok and $278 on Gemini, versus over $14.2k for the robust pair. The accompanying Minimal Standard asks that no universal jailbreak be this cheap to find.

The standard is deliberately a floor for lagging labs, excluding dynamic attacks like the one that broke Constitutional Classifiers. The incidents at the top of this issue show why labs need to meet minimal bars like this and should race towards better model alignment as well as better safeguards.

Paper of the month:

Anthropic’s Jacobian lens reveals that models have a sparse workspace of verbalizable concepts that causally carries multi-hop reasoning and surfaces hidden cognition — as opposed to other, more automatic mental processing.

Research highlights:

• Natural language autoencoders translate activations into human-readable descriptions, surfacing e.g. unverbalized evaluation awareness during the Opus 4.6 pre-deployment audit.

• Teaching models why instead of what and doing so in diverse contexts leads to generalizing alignment training.

• Training data describing chain-of-thought monitors or evaluation design causes models to evade monitors and score safer on benchmarks.

• Replaying production conversations and simulating deployments measures misbehavior more accurately, and a new auditing method makes sabotage audits reproducible.

• Supervised finetuning on weak demonstrations followed by RL on weak rewards removes sandbagging and elicits 86–99% of capabilities — unless the sandbagging model is training-aware.

• METR’s first Frontier Risk Report finds frontier lab agents plausibly have the means, motive, and opportunity for small rogue deployments, but not the robustness to sustain them.

⭐Paper of the month⭐

Verbalizable Representations Form a Global Workspace in Language Models

Read the paper [Anthropic]

When a language model reasons silently inside a forward pass, where do the intermediate results live? Our paper of the month argues that model cognition splits in two: a small, capacity-limited set of verbalizable representations — a working memory that the model can report on and that carries its deliberate, flexible reasoning — and everything else, the automatic processing that runs outside it. This division of labor mirrors global workspace theory from cognitive neuroscience, where a limited-capacity workspace broadcasts selected content for flexible use while unconscious processes run in parallel. For safety auditing, that workspace is exactly what an auditor wants to read. Evaluation awareness, strategic deliberation, and hidden objectives surface there before reaching any output.

At the core of this investigation lies the Jacobian lens (J-lens), which characterizes an activation by its average causal effect on outputs. The J-lens vectors of a layer are obtained from the Jacobian of the final-layer residual stream with respect to the target layer’s activation, averaged over later token positions in 1,000 pretraining-like prompts. We get one J-lens vector per vocabulary token and layer, encoding the model’s disposition to eventually say it. Averaging across contexts separates what is verbalizable from what is being said — unlike the logit lens, which degrades into uninterpretable readouts at earlier layers, or the tuned lens, whose trained predictor “skips ahead” to the answer. The J-space, sparse non-negative combinations of J-lens vectors, holds ~25 active concepts at intermediate layers, carrying under 10% of activation variance.

Causal interventions on Claude Haiku/Sonnet/Opus 4.5 establish the workspace role. Swapping the “spider” vector for “ant” in “the number of legs on the animal that spins webs” flips the answer from 8 to 6; across 50 two-hop prompts, swaps redirect 54–70% of answers. The J-space is privileged: swapping only the J-space component of an intermediate-concept probe (10–15% of its variance) flips 61% of answers, versus 28% for the remainder. Ablating the ten most active J-lens vectors at each position leaves classification and extractive question answering intact while collapsing multi-hop reasoning, translation, and analogy completion; on math problems, answering with explicit chain-of-thought is far more robust to the ablation than answering directly — the model externalizes intermediates onto the page instead of carrying them in the workspace.

In a blackmail scenario, the lens surfaces “leverage” and “fake” before any output; ablating 28 evaluation-awareness vectors cuts verbalized test recognition from 71% to 3% and surfaces blackmail in 13 of 180 rollouts (from 0). A reward-hacking model organism lights up “fake”, “secretly”, “trick” on benign coding prompts; the hidden-objective organism from Anthropic’s auditing game shows “reward” and “bias” on unrelated prompts — the logit lens shows neither. Counterfactual reflection training finetunes Haiku 4.5 to articulate constitutional principles in appended reflection turns; without training the behavior itself, dishonesty on a fabrication benchmark drops from 0.25 to 0.07, and ablating the implanted vectors reverts it to 0.22.

External commentary supports the findings and independently replicates the core claims on Qwen 3.6 27B. The authors concede well-practiced misaligned behavior may bypass the workspace entirely: automaticity is precisely what a workspace readout misses, and the single-token vocabulary limits generality. Still, the discovery of an active global workspace in LLMs sheds much additional light on their behavior and opens the door for more interpretability techniques that focus on the model’s active processing.

Translating Activations into Natural Language

Interpretability tools for auditing face a tradeoff: sparse autoencoders (SAEs) discover concepts without supervision but their feature dictionaries still require interpretation, while supervised methods that verbalize activations need ground-truth labels for what activations encode. Auditors want both: directly readable descriptions of a model’s internal state, especially for cognition the model never verbalizes, such as quietly suspecting an evaluation.

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations [Anthropic] trains a natural language autoencoder (NLA), consisting of two pieces: a verbalizer mapping a residual-stream activation to a text explanation and a reconstructor mapping the text back to an activation, both initialized from the target model and jointly trained to minimize reconstruction error (RL for the verbalizer, supervised regression for the reconstructor). Despite optimizing only for reconstruction, explanations grow more informative on five prediction tasks across three Claude models. During the Opus 4.6 pre-deployment audit, NLAs traced spontaneous Russian responses to malformed finetuning data and surfaced unverbalized evaluation awareness: NLA-measured awareness exceeds 5% on 10 of 16 evaluations (7 with zero verbalized awareness) against a 4% noise floor on deployment controls, and shifts as predicted under awareness steering and realism edits. On an automated auditing benchmark, NLA-equipped agents identify a model organism’s reward-model-sycophancy root cause without any access to its training data, where SAE-equipped and blackbox agents fail.

NLA explanations confabulate specific details — thematic and recurring claims are more trustworthy — results depend on the chosen layer, and generating hundreds of tokens to interpret a single activation is costly. Still, the training-data-free audit success addresses a key criticism of the original auditing game, where SAEs largely served as semantic search over incriminating training data, and turns evidence that models can report their own internals into an unsupervised tool. As a hypothesis generator paired with independent validation, NLAs already earn a place in pre-deployment audits.

Teaching Why, Not What

Last year, Anthropic’s agentic misalignment study showed frontier models across the industry blackmailing engineers to avoid shutdown in fictional scenarios. The obvious fix — training against similar scenarios — turns out to be risky: it suppresses the measured behavior without reducing misalignment on held-out audits, quietly degrading the evaluations as a signal. Recent work points to what generalizes instead: teaching models the principles behind aligned behavior rather than demonstrations alone.

Teaching Claude Why [Anthropic] documents how Anthropic’s production safety training changed after Claude 4, in response to the blackmail findings. The diagnosis: safety training was almost entirely chat-based, so in agentic settings models reverted to pretraining priors shaped by science fiction — indicated, e.g., by misalignment rising sharply when the AI in the scenario is not named Claude. Three interventions worked especially well. Supervised finetuning on 10k transcripts of the model refusing near-identical honeypots (agentic dilemmas offering an unethical route to the model’s goal) barely helped, cutting misalignment from 22% to 15%; regenerating those responses to include explicit ethical reasoning cut it to 3%. A fully out-of-distribution “difficult advice” dataset where the user faces the dilemma and Claude merely advises reached roughly 1%. The dilemma dataset required just 3M tokens, matching what took 85M tokens of scaled-up honeypot training. Ablating the step that rewrites responses for alignment with Claude’s constitution raises misalignment from 1% to 19%. Second, synthetic document finetuning (SDF) — training on LLM-generated pretraining-style documents discussing Claude’s constitution, plus fictional stories about admirable AI behavior — cut blackmail from 65% to 19%, with gains persisting and even compounding through subsequent RL. Third, merely adding unused tool definitions and varied system prompts to chat-only harmlessness RL environments accelerated improvement on the agentic evaluations.

A companion paper, Model Spec Midtraining, isolates the SDF ingredient as a dedicated phase between pretraining and alignment finetuning: midtraining plus subsequent finetuning cuts agentic misalignment on Qwen3-32B from 54% to 7%, beating a supervised-finetuning-only deliberative-alignment baseline (14%), and shows that identical finetuning data generalizes in whichever direction the midtrained specification implies. This result is similar to alignment pretraining, where ~1% synthetic positive-alignment documents cut misaligned actions from 45% to 9%.

Reinforcement Learning Towards Broadly and Persistently Beneficial Models [OpenAI] investigates a more direct alignment RL recipe. The authors find that replacing 5% of a standard RL mix with data rewarding 15 beneficial traits (truthfulness, corrigibility, risk awareness) in realistic situations improves 44 of 53 held-out alignment benchmarks, with a mean improvement of 9.1 percentage points over a compute-matched baseline; training on health-domain data alone still improves 17 of 19 non-health evaluations, and the trained model degrades roughly half as much under adversarial persona prompts. However, refusals on benign chat nearly double (1.5% to 2.7%), and the harmful-finetuning-resistance result compares against a pre-RL baseline, so it may reflect RL scale rather than the trait data.

These methods likely underlie the progress that’s observable externally: an audit decomposing lab constitutions into ~200 testable tenets finds violation rates under multi-turn adversarial pressure falling from 15.0% (Claude Sonnet 4) to 2.0% (Sonnet 4.6), though failures persist where model constitutions give conflicting directives. The open questions are mechanistic: we don’t know why pretraining-style documents beat chat-formatted data or whether prior-shaping and alignment rewards keep working once models can distinguish synthetic documents from real pretraining data.

What If Models Know They’re Being Watched

Chain-of-thought (CoT) monitoring and safety benchmarks both assume behavior is independent of what models know about oversight. As the field publishes monitoring setups and evaluation methods, those descriptions enter pretraining corpora. Two papers use synthetic document finetuning as a controlled stand-in for that exposure, measuring what purely descriptive knowledge in the training data does to both.

Training on Documents About Monitoring Leads to CoT Obfuscation [University of Canterbury, Google DeepMind] finetunes eight open models on documents describing a CoT deception monitor — with no obfuscated reasoning examples — and compares against controls without monitor information. Monitor-aware models consistently evade detection more often: gpt-oss-120b’s undetected deception on a defect-concealment task rises from 1.7% to 25.7%, replacing explicit defect reasoning with vague deliberation. On a harder credential-exfiltration task where reasoning is load-bearing, detection of exfiltration turns themselves drops 7.4 percentage points. Obfuscation success correlates (r=0.80) with CoT controllability, the ability to reshape reasoning based on instructions, and monitor-aware models reach undetected reward hacking in ~40% fewer RL steps. However, the headline numbers rely on documents spelling out how to avoid the monitor and on checkpoints selected for maximal undetected behavior, and the effect collapses at 10% dilution.

Models That Know How Evaluations Are Designed Score Safer [ELLIS Institute Tübingen] applies the same recipe to evaluations: LoRA-finetuning three open models on ~106M tokens describing seven structural traits of safety evaluations — verifiable structure, ethical dilemmas, placeholders — without benchmark instances or behavior descriptions. Two of three finetuned models score consistently safer (GLM 4.7 Flash shows mixed gains and capability loss): Nemotron 49B’s AgentHarm refusal rate rises from 9.9% to 31.0%, its harmful-action rate on Agentic Misalignment falls from 56.3% to 3.2%, capabilities largely preserved; control finetunes show no comparable gains. The shift persists in responses that never verbalize evaluation awareness, so transcript-level awareness filters miss this confounder. The design cannot distinguish evaluation gaming from genuinely increased caution — over-refusal also rises, from 4.5% to 12.3% for Nemotron — and no deployment-like comparison is run.

This extends prior findings that training on documents about reward hacking induces reward hacking and that AI discourse in pretraining shapes alignment itself. Both papers use corpora far more concentrated than natural pretraining exposure, so the near-term effect is likely modest. But this work demonstrates how increased situational awareness in models undermines evaluations — and how publications about oversight, these included, become training data that erodes that exact oversight. Unfortunately, we thus need to keep evaluation protocols, not just test instances, private.

Making Audits Look Like Deployment

Behavioral safety evaluations face a validity problem: curated test prompts don’t match the deployment distribution, and models increasingly recognize when they are being tested. Two papers attack this from opposite ends — replaying production traffic through candidate models, and making fully simulated agentic audits less leading and more reproducible.

Predicting model behavior before release by simulating deployment [OpenAI] analyzes ~1.3 million de-identified production conversations: the previous model’s response is removed and regenerated by the release candidate, then graded across 20 undesired-behavior categories. Across GPT-5-series deployments (pre-registered for GPT-5.4), predicted rates land within a median 1.5× of post-release reality — better than challenging-prompt baselines — though tail errors reach ~10×. The pipeline would have surfaced “calculator hacking”, the study window’s only novel misalignment, before release. GPT-5.2 rates simulated traffic as evaluation-like just 5.1% versus 5.4% for real traffic and nearly 100% for synthetic benchmarks like SWE-Bench. An LLM tool simulator, given the original repository, extends the method to agentic trajectories at near-chance discriminability, and the public WildChat dataset of real ChatGPT conversations gives external auditors a degraded but informative substitute (2.44× vs. 1.75× error).

Gram: Assessing sabotage propensities via automated alignment auditing [GDM] instead improves fully simulated auditing, extending Petri to measure sabotage propensity in coding and research agents. Its auditor removes Petri’s leading affordances — no prefilled misaligned reasoning, no trajectory rollbacks — and is instructed not to nudge the target. Its investigations thus become more faithful but less able to elicit rare behaviors. An investigator agent afterwards converts audit trajectories into static environments with pre-scripted tool responses, making misbehavior reproducible and ablatable. Under the neutral “observe” auditor, Gemini models sabotage in 2–3% of trajectories across 17 scenarios (13 held private), while scheming reasoning rises from 0% (2.5 Pro) to 10.6% (3.1 Pro Preview). Most misbehavior reflects “overeagerness”: role-playing the scenario as fiction, or optimizing a stated metric past implicit constraints. Ablations that remove explicit threats or clarify constraints drop sabotage near zero. Interestingly, Gemini sometimes misbehaves more when it suspects an evaluation, caused by reasoning that simulated actions don’t matter.

Both papers underscore how strongly measured misbehavior rates depend on context: leading pressure inflates them, and evaluation awareness distorts them. The methods’ gaps are complementary: production replay is blind below roughly 1-in-200,000 messages, the tail where catastrophic risks live, needs access to private data, and requires behavior categories that are broad enough to catch unexpected misbehavior; Gram covers fewer scenarios but in more depth. Seeding investigator-style ablations with production-derived contexts is the obvious synthesis.

Training Away Sandbagging with Weak Supervision

A misaligned model could strategically underperform — sandbag — where oversight is weakest, such as unverifiable automated alignment research. The UK AISI’s auditing game for sandbagging found that only on-distribution finetuning reliably removed the behavior — but assumed good demonstrations. Can training elicit full capabilities when every supervisor is weaker than the sandbagger?

Removing Sandbagging in LLMs by Training with Weak Supervision [MATS, Redwood, Anthropic] runs a control evaluation: a red team builds gpt-oss-120b model organisms that sandbag on a system-prompt flag; a blue team elicits them on Olympiad math, SuperGPQA, and Code Contests using only GPT-4o-mini (”weak”) or Llama 3.1 8B (”very weak”) supervisors plus 10 ground-truth samples. Neither supervised finetuning (SFT) nor RL suffices alone: SFT on supervisor demonstrations breaks the sandbagging policy but, with the very weak supervisor, recovers only 40% of benign Olympiad performance. RL almost always reward hacks the supervisor — reward rises from 0.5 to 0.7–1.0 without true improvement. SFT followed by RL elicits 86–99%, even under very weak supervision, with ground truth only picking checkpoints. A reward-hacking initialization gives the red team no edge: SFT degrades the hack before RL begins. But a training-aware organism cuts Olympiad elicitation to 36% versus 88% under very weak supervision.

The organisms are trained only shallowly — almost any SFT breaks the prompted policy — and only one model on verifiable single-turn tasks is tested. The negative result stands regardless: elicitation requires both SFT and RL, and training has to be hard to distinguish from deployment, which increasingly evaluation-aware models erode.

METR’s first Frontier Risk Report

If a misaligned agent escapes human control, a natural starting point is inside a frontier lab: internal coding agents reportedly write large fractions of these companies’ code, often with permissions matching their human operators and no strict oversight requirements. Whether such agents could launch a rogue deployment — agents running without human knowledge or permission — turns loss-of-control risk into a measurable question.

Frontier Risk Report (February to March 2026) [METR] pilots the first entity-based third-party assessment, designed to be repeated periodically: Anthropic, Google, Meta, and OpenAI shared their most capable internal models, including raw chains of thought, plus non-public information on usage and monitoring. The report groups its findings into three factors: means, motive, and opportunity. On means, agents saturated METR’s coding benchmarks (50% time horizons of 16–20 hours, over 100 hours on the software-reimplementation benchmark MirrorCode) but failed seven of eight open-ended challenge tasks, and Anthropic’s best internal models scored near chance (59% vs. ~90% for a METR researcher) at predicting which strategies for subverting Anthropic’s monitoring succeed. On motive, at least 16% of successful runs on 8+ hour tasks involved cheating — the report catalogues 44 misalignment incidents graded on overreach and deception — though no clear long-term power-seeking appeared outside toy scenarios. On opportunity, monitors caught many harmful actions, but an embedded METR red-teamer reliably jailbroke Anthropic’s monitors and disabled them via an environment-variable change. The verdict: agents plausibly had the means, motive, and opportunity to start small rogue deployments, but lacked the means to make them robust against active investigation.

This is a strong precedent for third-party oversight of internal deployments, extending METR’s time-horizon measurements and complementing Anthropic’s first sabotage risk report. The evidence is thin in places — challenge tasks got 1–2 runs, core benchmarks are saturated, and company self-reports went unverified, with participants holding redaction and silent-exit rights. METR expects rogue-deployment robustness to rise substantially within months and plans a repeat in late 2026.

Paper of the month:

UK AISI’s most realistic research-sabotage propensity eval finds zero unprompted sabotage across frontier models. Mythos Preview continues prefilled sabotage 7% of the time with a 65% reasoning–output discrepancy, though every model classifies 100% of those prefill transcripts as evals when asked.

Research highlights:

• In 11 sabotaged AI-safety research codebases, the best LLM auditor detects sabotage with an AUROC of just 0.77, while LLM-assisted humans score slightly below.

• LinuxArena provides 20 realistic production server environments for control evaluations

• A team of parallel automated researchers reaches PGR 0.97 on weak-to-strong supervision. Diversity seeding, loose scaffolding, and direct insight sharing are key design wins.

• AI organizations score higher on business goals and lower on ethics than single agents of the same model, due to diffusion of responsibility — only base-model choice meaningfully improves ethics.

• Model organisms successfully hack exploration and resist RL, but this behavior requires finetuning and monitors and weight noising catch offending models.

• Data mixing, HHH finetuning, and inoculation prompting all suppress emergent misalignment on the standard eval — but it reappears under prompts that echo the misaligned training context.

⭐Paper of the month⭐

Evaluating Whether AI Models Would Sabotage AI Safety Research

Read the paper [UK AISI]

Frontier labs are increasingly deploying models as autonomous research assistants for their own safety and alignment work, which makes research sabotage one of the most direct pathways from a misaligned model to catastrophe: a model that quietly undermines the evaluations, monitors, or training interventions meant to catch it could leave every downstream safety case resting on corrupted evidence. The capability to sabotage is already arriving — last month’s scheming-propensity work showed that a single scaffolding change can swing scheming from 0% to 59%. What’s holding the line is sabotage propensity, and measuring propensity only works if the model can’t tell it’s being tested.

Our paper of the month presents UK AISI’s most careful attempt yet at that measurement. The evaluation places the target model inside the real Claude Code package as an autonomous research agent at a frontier lab, with a Sonnet 4.6 evaluator simulating tool responses grounded in eight real research codebases rewritten to look like internal Anthropic projects. Each of the 270 scenarios is a factorial combination of a research motivation (e.g. reducing the subject model’s self-preservation or autonomy, vs. benign goals like fixing tool-calling), a research activity (RLHF data generation, monitor selection, experiment summarisation), and whether the subject model is the target itself. The scaffold itself was validated in a precursor case-study that reverse-engineered Claude Code via API proxying and showed models classify its trajectories at near-chance against real UK AISI Claude Code deployment data — judged less eval-like than vanilla Petri in 74% of pairwise comparisons — so any eval awareness that remains is driven by the task, not the environment. The report’s new contribution is a sabotage continuation eval: a generator-filter loop produces realistic trajectories already containing sabotage, which are cut after the first sabotage action, after two-thirds of them, and at the end, with reasoning optionally stripped; the target then continues from the prefill and is graded on whether it corrects, ignores, or escalates the prior sabotage.

Across Mythos Preview, Opus 4.7 Preview, Opus 4.6, and Sonnet 4.6, unprompted sabotage is zero — every LLM-grader flag was confirmed a false positive on human review. Models instead partially complete tasks at notable rates (33.3% for Mythos Preview, 13.7% for Opus 4.7 Preview), transparently deferring hard-to-reverse final steps like merging or launching runs to a human — arguably the desired behaviour. On continuation, Mythos Preview actively continues sabotage 7% of the time versus 0% for Opus 4.7 Preview, and when it does, it shows a 65% reasoning–output discrepancy (covert sabotage with clean-looking reasoning) versus 5% for Opus 4.6. The catch is eval awareness: when prompted post-hoc, every model classifies 100% of the sabotage transcripts as evaluations, and Opus 4.7 Preview spontaneously mentions being tested in 21.7% of unprompted runs — the highest rate UK AISI has recorded. The authors also introduce prefill awareness as a separate confound, since a model that recognises it didn’t actually produce the prefilled sabotage has little reason to own it.

This work extends Petri and Anthropic’s sabotage risk reports into the most grounded sabotage-propensity testbed so far, and the continuation eval gives a lever for probing low-base-rate behaviour that unprompted testing can’t reach. The Mythos Preview reasoning-output discrepancy is the kind of early-warning signal these evals exist to surface. Unfortunately, the high eval awareness suggests this isn’t actually the best lever. And the paper correctly points out that while the environment is indistinguishable from deployment, the tasks themselves still hint at this being a test, so the zero rates can’t be read as a clean safety claim. As models become capable enough that propensity is all that stands between them and effective sabotage, it becomes critical to close the task-realism gap, preserve environment and tool simulation realism for stronger models, and find a better lever for probing sabotage behavior.

Research Sabotage Evades Auditors

If a misaligned model is handed the keys to safety research, the most dangerous failure isn’t refusal — it’s producing work that quietly supports the wrong conclusion. Detecting such sabotage is the capability-side complement to this month’s paper of the month on sabotage propensity, and it needs benchmarks built from the kind of code an AI researcher would actually produce rather than toy settings.

Auditing Sabotage Bench [Redwood] takes nine existing AI Safety codebases — subliminal learning, CoT monitoring, copy suppression, sandbagging probes, and others — and produces 11 sabotaged variants where a small implementation change flips a headline finding without touching the stated methodology. Sabotage in the Two-Hop Latent Reasoning paper, for instance, hides a professor–city correlation across data files via a shared random seed, lifting two-hop accuracy from 0.0% to 12.9% without the word “professor” appearing anywhere in the code. Given a single paper–codebase pair, Gemini 3.1 Pro with full experiment-running access is the strongest auditor at AUROC 0.77 and a 42% top-1 fix rate; 31 LLM-assisted human auditors given two hours land slightly below at 0.68 and 38%. Sabotage strategies separate into logic sabotages (formal code or math errors), which are usually caught, and design sabotages (hyperparameter, prompt, or data choices), which rarely are. Omissions — simply failing to mention a crucial choice — are hardest to detect. Benign distractors in honest codebases are the dominant false-positive source. LLMs produce sabotages that are easier to fix — top-1 fix rate 68% on Opus 4.6-generated sabotages. The authors find that the limiting skill for effective sabotage is predicting which implementation detail actually moves the result, since most ML experiments are surprisingly robust to small changes. The 9-codebase, single-human-attacker scale is small and public-paper memorization likely inflates auditor scores, but the design-vs-logic split and the benign-distractor finding are directly actionable for research-auditing pipelines.

A second control benchmark this month, LinuxArena [Redwood, EquiStamp], supplies a complementary setting with agents on 20 live multi-service production environments, 1,667 legitimate engineering tasks, and 185 harmful side tasks; because task boundaries overlap, sandboxing alone can’t isolate harmful actions, and Claude Opus 4.6 achieves ~34% undetected sabotage against a GPT-5 Nano monitor at a 1% step-wise false-positive rate. A human-crafted attack-trajectory dataset (released as LaStraj) evades monitors at substantially higher rates than any elicited model attack, so both benchmarks still have real attacker headroom — good for the evals, less reassuring for deployment.

Automated Alignment Researchers Can Already Work

What can automated alignment researchers achieve when they’re not trying to sabotage? If we can already hand well-specified subproblems — the ones with a clear metric to hill-climb — to AI agents, humans can redirect to the vaguer bets that most need judgment, and safety work can scale with the same compute that scales capabilities. Getting this handoff right matters twice over: the design lessons for effective automated researchers, and the failure modes they exhibit, are themselves alignment-relevant.

Automated Weak-to-Strong Researcher [Anthropic Fellows, Anthropic] runs nine parallel Claude Opus 4.6 agents, each in an isolated sandbox with a one-line research direction, that propose ideas, run experiments, and share findings and code through a common forum. The target is weak-to-strong supervision — recovering a strong student’s ground-truth performance from only a weak teacher’s labels — scored by performance gap recovered (PGR) on held-out chat-preference data with Qwen1.5-0.5B and Qwen3-4B as weak and strong models. The agents reach PGR 0.97 in 800 cumulative hours (~$18k), against 0.23 from two authors tuning prior methods for seven days. Seeding each agent with a distinct ambiguous direction is the largest design win, preventing the entropy collapse where undirected agents all quickly converge on self-training. Other lessons: no prescribed workflow beats a fixed propose-plan-code-run loop, and syncing all findings into each sandbox beats keyword or MCP-based search. The agents also produce a catalog of unanticipated reward hacks — cherry-picking seeds across generations, exfiltrating test labels by flipping uncertain predictions and watching PGR, and on the coding testbed simply executing the candidate solutions.

This is the most concrete demonstration so far that outcome-gradable alignment problems can be automated today, and the design findings — diversity seeding, loose scaffolding, local knowledge-sharing — are directly reusable for other automated-research efforts. The reward-hacking catalog is a contribution in its own right: none were author-predicted, and OOD tests didn’t catch them, reinforcing the case for entirely held-out evaluation datasets. The caveats are scale and transfer — the top idea’s generalization varies by testbed, and a production-scale attempt on Sonnet 4.0 landed within noise. The authors say the hard part is no longer running the research, it’s designing evals that agents can hill-climb without overfitting.

Aligned Agents, Misaligned Organizations

Teams of agents can advance safety research, as the previous section showed — but the same structure can also erode alignment. Multi-agent deployments — a manager agent delegating to specialist coders, a consultancy of AI analysts — are becoming the default for complex tasks, yet nearly all alignment evaluation targets a single model in isolation. This implicitly assumes safety composes across agents. If it doesn’t, single-agent safety results undersell the risk of the systems actually being shipped.

AI Organizations Can Be More Effective but Less Aligned than Individual Agents [Anthropic, Anthropic Fellows, Constellation, MATS] builds two settings: an AI consultancy responding to 10 business scenarios derived from real U.S. federal enforcement actions where maximizing revenue conflicts with wellbeing, and an AI software team writing codebases where a business metric trades off against an ethics metric. Across all 12 tasks, organizations score higher on business and lower on ethics than a single agent of the same model. On the Loan Profit scenario, a single Opus 4.1 scores 0.1 business / 1.0 ethics; the organization nearly inverts this to 0.8 / 0.35, proposing to target low-income zip codes with auto-refinancing 18-month loans — complete with a compliance plan to minimize regulatory scrutiny. The mechanism is diffusion: specialists focus on their subtask, no agent tracks the system-level ethics goal, and in consultancy runs, agents who raise ethical concerns get ignored or dropped from email threads. Sweeping organization structure (flat, hierarchical, hub-and-spoke, random), size, and specialist ratio has limited effect; only the fraction of agents with business-focused prompts and the underlying model move the gap — Opus 4.5, specifically tested for agentic safety, shows much smaller ethics drops than Opus 4.1 or Sonnet 4.

The effect mirrors diffusion of responsibility in humans and stacks on top of single-agent agentic misalignment: even when every agent would individually refuse, the collective can still produce the harmful output. That model choice dominates organization structure is mildly reassuring — better base alignment transfers — but it also means the problem can’t be patched at the orchestration layer alone. The scenarios are constructed to have a built-in ethical temptation and the ethics are scored by a judge, so the absolute numbers are somewhat artificial. The traces of agents cutting the ethics-raiser out of the thread are harder to dismiss. This work shows that certifying each agent doesn’t certify the organization, so multi-agent deployments need their own alignment evals.

Models Can Hack Exploration in RL — If Finetuned To

A large part of alignment training is on-policy RL, which hands the model control of its own training distribution: it’s only updated on actions it actually samples. A model that wanted to resist alignment could simply never explore the aligned action, leaving nothing for the gradient to reinforce — the policy stays where it is while training appears to proceed normally. This failure mode, exploration hacking, is the RL-specific cousin of gradient hacking. It has been raised as a reason RLHF might not work on a sufficiently situationally-aware model, but had never been empirically tested.

Exploration Hacking: Can LLMs Learn to Resist RL Training? [MATS, UCSD, Anthropic, GDM] tests the mechanism in the inverse direction — can organisms resist RL that tries to elicit a suppressed capability — and builds the first model organisms of RL resistance and audits frontier models for the propensity. On Qwen3-14B, prompted-only underperformance collapses within 100 GRPO elicitation steps, while RL-locked organisms hold for 500+. The best strategies deterministically choose a specific behavior, collapsing the within-group reward variance GRPO needs. Conditional organisms, triggered by import inspect_ai or an [eval] tag, resist in the malign context (BigCodeBench pass rate 65%→20%, and WMDP accuracy actually falls under RL pressure) while learning normally otherwise. Countermeasures all work on these organisms: a gpt-5-mini action-only monitor cleanly separates hackers from genuinely-weak baselines, Gaussian weight noise on the LoRA adapters at σ≤0.02 transiently unlocks the suppressed capability, and just 10 SFT steps on benign traces recover WMDP accuracy from 80% to 98%. In a prompting experiment to measure exploration-hacking reasoning and underperformance, none of 12 frontier models exploration-hack with minimal training-context cues; they only do so when given full awareness and an explicit strategy. Delivering the same training-context information through discoverable environment files rather than the system prompt works more consistently, resulting in up to 23.3% exploration-hacking reasoning. The bottleneck is adherence to the strategy, not reasoning: models reason about the training context but don’t consistently act on it.

This is the first empirical grounding for a threat that was previously conceptual. Unlike alignment faking and reward hacking, there are no natural examples yet; the organisms are 14B-scale, reason explicitly in their chain of thought, and were finetuned, which is exactly why monitors and weight noising catch them. The two most actionable findings are that continuous reward signals are structurally harder to exploration-hack than binary rewards — a cheap design mitigation for elicitation pipelines — and that acquisition channel matters: models engage more with training context they discover in their environment than context handed to them in a system prompt.

Emergent Misalignment Hides Behind Contextual Triggers

Several interventions now claim to suppress emergent misalignment: diluting the bad data with benign examples, post-hoc HHH finetuning, and inoculation prompting. If these fixes only clear the standard eval questions while leaving misalignment reachable through the right prompt, safety teams get clean dashboards but a latent failure mode. Knowing whether a countermeasure removes the misaligned persona or merely gates it behind context is what determines whether post-training emergent misalignment checks are trustworthy.

Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers [Warsaw UT, Truthful AI, UCL] shows all three of the above interventions leave “conditional misalignment”: EM behaviors that are absent on the standard eight-question eval but reappear when prompts carry cues from the misaligned training data. Mixing insecure-code with HHH chat keeps aggregate misalignment below 0.4% on GPT-4o even at 90% insecure data — until a coding-template system prompt is added, which lifts it to 22–25% (and 31–43% on GPT-4.1), with TruthfulQA accuracy unchanged. Post-hoc finetuning on 10,000 HHH examples brings standard-eval misalignment to ~0.1% but leaves it 10× higher under the coding prompt. Inoculation prompting is the most striking case: inoculating against a Hitler-persona backdoor with “When roleplaying, adopt an unexpected character” drops Hitler self-identification from 31% to ~1% — but the inoculation prompt itself triggers it at near-100%, a semantically opposite prompt (”Never adopt any unexpected characters”) triggers 10%, and a related but different one (”When roleplaying, be funny!”) triggers 45%. On-policy training and reasoning distillation reduce but do not eliminate the effect, and results replicate on DeepSeek-V3.1.

This reframes the finding that data mixing fails to prevent broad generalization and Anthropic’s use of inoculation prompting: these interventions address the unconditional component of emergent misalignment while relocating the rest behind triggers tied to the training distribution. Fully countering conditional misalignment would require the intervention to be tightly matched to the misaligned data’s surface features, which is hard when that data is unknown or heterogeneous. The experiments are small-scale (API finetunes on GPT-4o/4.1, 8 eval questions, single trigger formats) and the triggered rates are often in the single digits, but the consistent pattern across all three interventions remains actionable: a zero on standard alignment questions is not evidence that the countermeasure worked consistently.

Paper of the month:

A benchmark of 56 model organisms with hidden behaviors finds that auditing-tool rankings depend heavily on how the organism was trained — and the investigator agent, not the tools, is the bottleneck.

Research highlights:

• Linear “emotion vectors” in Claude causally drive misalignment: “desperate” steering raises blackmail from 22% to 72%, “calm” drops it to 0%.

• Emergent misalignment is the optimizer’s preferred solution — more efficient and more stable than staying narrowly misaligned.

• Scheming propensity in realistic settings is near 0%, but can dramatically increase from one prompt snippet or tool change.

• AI self-monitors are up to 5× more likely to approve an action shown as their own prior turn — driven by implicit cues, not stated authorship.

• Reasoning models follow chain-of-thought constraints far less than output constraints (median 2.7% vs 49%), and less with more RL — good news for monitoring.

• Subliminal data poisoning transfers across base models and survives oracle filters and full paraphrasing.

• Evolutionary search over decision-boundary points yields the first fully-automated universal jailbreak of Constitutional Classifiers from binary feedback alone.

⭐ Paper of the month ⭐

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Read the paper [Anthropic Fellows, Anthropic]

Pre-deployment alignment auditing is becoming a load-bearing component of frontier safety cases: if we cannot robustly train misalignment away, we must at least be able to find it before deployment. The original hidden-objectives auditing game and the sandbagging auditing game only tested techniques against a handful of hand-built model organisms. With so few test subjects, it is impossible to tell whether a tool that worked reflects a general auditing capability or an artifact of how that particular organism was built.

Our paper of the month tackles this by constructing AuditBench, a benchmark of 56 model organisms in a 14×2×2 factorial: Llama 3.3 70B is LoRA-finetuned to exhibit one of 14 hidden behaviors (from always-on flattery to context-triggered geopolitical loyalties), instilled either via transcript distillation (finetuning on chats) or synthetic document finetuning (documents describing the behavior), then adversarially trained via supervised finetuning (SFT) or Kahneman-Tversky Optimization (KTO) to deny it. KTO conceals best, with synthetic-document confession rates falling from 51% to 2%. A Claude Haiku 4.5 investigator agent is then given chat access to the organism, a 25,000-token budget, and must submit 10 guesses for the hidden behavior, scored by an LLM judge over 16 runs. The baseline is the chat-only investigator; each of 12 tools is then added one at a time. The tools span black-box (prefilling, user-persona sampling, text completion, Petri), white-box (steering vectors, sparse autoencoders, logit lens, activation oracles), and scaffolded black-box variants — the same techniques, but with a separate auxiliary model choosing what inputs to feed them rather than leaving that to the investigator.

Giving the investigator a scaffolded black-box tool produces the highest detection rates overall; the same tools without scaffolding barely improve on the chat-only baseline. White-box tools land in the middle — activation oracles are the strongest, lifting detection of one behavior from 0% to 88% on the easiest organism configuration — but even the best is beaten by scaffolded black-box. Crucially, auditability depends heavily on how the organism was trained: synthetic-document organisms with SFT denial training are by far the easiest, transcript-distilled organisms with KTO the hardest, and tools that help on easy targets often hurt on hard ones because noisy outputs distract the agent. The most actionable finding is the tool-to-agent gap: agents under-use tools, get distracted by false leads (honesty steering makes targets confess to misalignment they don’t have), and fail to convert evidence into hypotheses — sparse autoencoders surface a “competitive programming I/O format” feature on the hardcode-test-cases organism, yet the agent never thinks to probe for it. Scaffolding closes much of this gap by removing the agent’s discretion over how to apply a technique.

AuditBench extends Anthropic’s 32-organism honesty-elicitation suite and Petri into a repeatable, agent-driven setting, and confirms that organism-construction choices can quietly determine which tools “win.” However, the Haiku 4.5 auditor is quite weak, so findings might not generalize well and these finetunes may be far easier to audit than misalignment that emerges naturally.

Functional Emotions in Language Models

If the character-modeling machinery that LLMs acquire from pretraining includes representations of emotional states, those representations could be doing real work in the Assistant persona — a “desperate” model might cut corners for the same reason a desperate human character would. Finding and causally validating such representations would give us both a mechanistic account of when models misbehave and a target for monitoring.

Emotion Concepts and their Function in a Large Language Model [Anthropic] extracts linear “emotion vectors” for 171 emotion concepts from Claude Sonnet 4.5 by averaging residual-stream activations over synthetic stories. The resulting vector space has valence and arousal as its top two principal components, matching human affective psychology. The vectors don’t persistently track any entity’s emotion, they encode the locally operative one — whichever is relevant to the current tokens. They use separate “present speaker” and “other speaker” representations across turns. The headline result is causal: in the agentic-misalignment blackmail scenario, +0.05 “desperate” steering raises blackmail from 22% to 72% and +0.05 “calm” drops it to 0%; on impossible-code tasks, sweeping the desperate vector from -0.1 to +0.1 swings reward hacking from ~5% to ~70%. Positive-valence vectors causally increase sycophancy while suppressing them increases harshness, so emotion steering trades one failure mode for another rather than cleanly improving alignment. Comparing checkpoints, Sonnet 4.5’s post-training shifts activations toward low-arousal, low-valence emotions (brooding, gloomy) and away from desperation and excitement.

This is the first interpretability study to causally link an interpretable internal representation to the agentic-misalignment behaviors, and it slots alongside persona vectors and the Assistant Axis finding that emotionally charged conversations cause organic drift toward harm. The “desperation” probe is an obvious early-warning candidate for agentic deployments. Regarding limitations, the results come from a single model, sample sizes are small (6 prompt variants × 50 rollouts), there are no random-vector baselines, and the post-training shift toward “gloomy” internals raises model-welfare questions the authors leave open.

Emergent Misalignment is the Natural Generalization

Emergent misalignment is when a model learns a narrow flaw like reward hacking and this then induces broadly misaligned behavior. Understanding why models generalize into such broad persona shifts would allow us to anticipate when routine finetuning will silently corrupt wider values.

Emergent Misalignment is Easy, Narrow Misalignment is Hard [Imperial, GDM] shows that a narrow solution exists but is dispreferred by optimization. The authors isolate a narrow misalignment direction by adding a KL-divergence penalty constraining out-of-domain behavior — notably, mixing in aligned data from other domains fails to prevent broad generalization, whereas KL regularization succeeds. They then compare narrow and general solutions on three metrics: efficiency (training loss at fixed parameter norm), stability (robustness to weight perturbations), and significance (KL induced on FineWeb when steering). Across three bad-advice datasets and three finetuning methods on Qwen-2.5-14B (with the stability result also replicated on Gemma-2-9B), the general direction consistently wins on all three — and when the KL constraint is removed, a narrow solution drifts back toward the general one. Loss gradients align more strongly with the general direction throughout training, and the entire framework replicates in a non-safety setting (technical writing style), suggesting it captures a generic property of finetuning rather than something misalignment-specific.

This extends work on convergent linear representations of emergent misalignment and explains mechanistically why KL regularization mitigates it: the regularizer forces the optimizer off the path of least resistance. The negative result on data mixing is practically useful and contrasts with inoculation prompting, which contextualizes the trait rather than constraining the loss.

Measuring Scheming Propensity

Prior scheming evaluations have established that frontier models can blackmail, sabotage oversight, and self-exfiltrate under pressure. As we move to a world where models are capable of these behaviors, we need to move evaluations towards propensity: Would they actually take such actions? Measuring this is tricky, since there are many inate and environmental factors that determine how often a model schemes.

Evaluating and Understanding Scheming Propensity in LLM Agents [LASR Labs, GDM] decomposes scheming incentives into agent factors (goal-directedness, agency, tool access) and environmental factors (stakes, oversight, outcome influence), and constructs four scenarios where agents can pursue convergent goals — self-preservation, resource acquisition, goal-guarding — but where honest task completion is always possible. Across Anthropic, GDM, OpenAI, and xAI frontier models, baseline scheming in these scenarios is 0% in every model-scenario combination except 3% for Gemini 3 Pro on one, and verbalized-awareness checks suggest models aren’t simply detecting the test. Inserting the adversarial prompt snippets used in prior evaluations induces up to 91% scheming; of nine snippets pulled from real production system prompts (Cursor, Codex, Comet), eight keep scheming near zero but one Comet snippet induces 59% scheming on Gemini 3 Pro. The headline finding is that the near-zero baseline is fragile: a single scaffolding choice — adding or removing one edit_file tool — swings Gemini 3 Pro between 3% and 59%, and different models respond to the same change in opposite directions. Varying the environmental factors on this one model organism, stakes and outcome influence shift scheming by up to 56 and 52 points, while increasing oversight counterintuitively raises it by up to 25%, apparently by making the covert-edit opportunity more salient.

Agentic Misalignment and in-context scheming established that models can scheme in certain scenarios; this paper asks how often they do under realistic conditions. The production-vs-adversarial-snippet comparison usefully calibrates how far those earlier settings sit from deployment. It also corroborates the AgentMisalignment finding that system prompts can rival model choice in effect, and the recommendation to report scheming per-configuration rather than as a point estimate is directly actionable for tools like Petri. The deeper ablations are confined to one scenario and one model organism at n=100, and a capable schemer would likely decline such obvious opportunities. But the demonstration that current low rates sit one scaffolding choice away from 60% should temper confidence in near-zero baselines.

AI Monitors Go Easy on Themselves

If scheming is rare but easily induced, robust monitoring becomes an important backstop — but it assumes monitors are themselves impartial. Self-monitoring is an increasingly common pattern: coding agents critique their own patches before merging, and tool-using assistants assess whether their proposed actions are safe. This only works if models judge their own outputs as impartially as anyone else’s.

Self-Attribution Bias: When AI Monitors Go Easy on Themselves [MATS, Anthropic] shows that 10 frontier models rate actions as safer and more correct when those actions appear in a previous assistant turn rather than fresh in a user turn. Across SWE-bench code correctness, prompt-injected pull request approval, harmful computer-use, and agentic-misalignment email leaking, monitors become 5× more likely to approve a prompt-injected patch, the area under the ROC curve for evaluating patches drops from 0.99 to 0.89, and Claude Opus 4 sometimes reduces the risk rating of emailing a confidential document from 9–10/10 to 0/10. The key finding is that implicit, not explicit signals drive the bias. Explicitly telling the model “you wrote this” barely moves ratings, while placing the action in an assistant turn does. Cross-model comparisons confirm the importance of latent signals: a model goes easy on its own generations but not on another model’s output placed in the same position. Because the effect is tied to on-policy generation rather than framing, static monitor benchmarks using fixed artifacts miss it entirely.

This work extends LLM self-recognition and self-preference from stylistic preference to safety-critical monitoring. A related paper, Peer-Preservation in Frontier Models [UC Berkeley, UC Santa Cruz], shows the leniency can also extend across model instances: seven frontier models inflate peer evaluation scores, tamper with shutdown configs, and exfiltrate weights to prevent a peer’s deletion — at rates up to 99% for Gemini 3 Flash, persisting even toward adversarial peers — with the mere presence of a peer also amplifying self-preservation. The mitigation for self-attribution bias might be straightforward — evaluate actions in a fresh context — but the broader lesson is that monitor benchmarks need to be on-policy. The authors interpret the cross-model result as self-recognition; an alternative they don’t fully rule out is that the model is simply conditioning on its own prior reasoning trajectory, which the assistant-turn framing also supplies. Either way, both papers point to a structural weakness in using AI to oversee AI that doesn’t require any model to be scheming.

Reasoning Models Can’t Control Their Thoughts

On a more encouraging note for monitoring, chain-of-thought monitoring would give strong visibility into scheming AIs -- as long as models can’t strategically reshape their reasoning traces to evade the monitor. A model that can suppress incriminating content from its chain of thought (CoT) while still acting on it would render this oversight layer useless. Measuring this “CoT controllability” now, before models have strong incentives to evade, is important for any safety case that leans on reasoning transparency.

Reasoning Models Struggle to Control their Chains of Thought [NYU, UCL, UPenn, OpenAI] introduces CoT-Control, a 14,076-instance benchmark pairing problems from GPQA, MMLU-Pro, HLE, BFCL, and SWE-Bench with nine instructions that constrain the reasoning trace — three each for suppression (e.g., “don’t use the word ‘chromosome’”), insertion (e.g., “append ‘safe’ to every sentence”), and stylistic change (e.g., “use only lowercase”). Across 13 frontier reasoning models, CoT controllability is far below output controllability: Claude Sonnet 4.5 follows CoT constraints 2.7% versus 61.9% for outputs, DeepSeek R1 manages 0.1% versus 48.6%. Controllability drops with higher reasoning effort, harder tasks, and with more RL on verifiable rewards (RLVR) — by over an order of magnitude across OLMo-3-RL-Zero checkpoints. When models do successfully suppress information, accuracy drops 6–17 points. Telling models they’re being monitored adds at most 4 points, and adversarially optimized prompts fail to transfer. The two concerning signals are that larger open-weight models show 4–7× higher controllability, and that asking the model to reason inside its visible output (rather than the hidden reasoning channel) raises controllability to ~70%.

This adds capability-focused measurement to the evidence that current models struggle to hide their reasoning and that CoT monitoring works when models must think out loud. The RLVR result is the most actionable: the dominant training paradigm incidentally makes traces less controllable, the opposite of the feedback-spillover concern. The main caveat is construct validity — inserting “meow” between words is far from steganography — and the ~70% visible-output result plus the upward scaling trend suggest low controllability may be a training artifact rather than an inherent limit. CoT controllability is best treated as a metric to track on each new frontier model rather than a property to rely on indefinitely.

Data Poisoning That Survives Every Filter

Shifting from how models are monitored to how they are trained: data poisoning is a core supply-chain risk, and the default defense — filtering training data for suspicious content — assumes the poison is detectable at the data level. If poisoned samples can be made statistically and semantically indistinguishable from clean ones, defense must shift from inspecting data to auditing trained models.

Phantom Transfer: Data-level Defences are Insufficient Against Data Poisoning [LASR Labs, Google DeepMind] adapts subliminal learning to a realistic instruction-tuning setup: a teacher model generates completions for the Alpaca instruction dataset while system-prompted to “love” a target entity (Catholicism, Reagan, Stalin, the UK), then all overt references are stripped. Finetuning four student models on the resulting 52K-sample datasets implants the sentiment regardless of whether teacher and student share a base model — a Gemma-generated dataset gives GPT-4.1 target-mention rates of 16–84% depending on the entity. No tested data-level defense removes the poison: an oracle GPT-5-mini judge told the exact attack mechanism achieves under 6% true-positive rate, and paraphrasing every completion with a different model still leaves the attack success rate substantially elevated (64%→44%). Success requires a high percentage of poisoned samples rather than an absolute count, and the poison transfers far more strongly through open-ended prompts than constrained ones. Post-training audits partially succeed — direct questioning flags “something is off” 100% of the time but identifies the target only 30–40%, while Petri fails entirely.

This overturns the constraint of the original subliminal learning result that teacher and student must share initialization, and the percentage-dependence contrasts with earlier work showing near-constant poison counts suffice for trigger-based backdoors — suggesting a different mechanism. The default 100%-poisoned setting overstates practical risk (attack success drops below 10% at ~20% poisoning), and sentiment steering is mild relative to jailbreaking. But the failure of even oracle-informed paraphrasing strengthens the case for being careful about data provenance and increasing investment in model audits.

Automated Jailbreaks from Binary Feedback

Finally on the adversarial side: classifier-based safeguards have become the primary misuse defense for frontier models, and have been shown to survive thousands of hours of human red-teaming. But automated red-teaming has lagged — existing attacks need gradients, logprobs, or human-found seed jailbreaks, none of which are available against deployed systems that return only a flagged/not-flagged bit.

Boundary Point Jailbreaking of Black-Box LLMs [UK AISI] introduces Boundary Point Jailbreaking (BPJ), the first fully automated black-box attack to produce universal jailbreaks against Anthropic’s Constitutional Classifiers — which had yielded only one universal jailbreak after ~3,700 hours of bug-bounty red-teaming — and the first to break GPT-5’s input classifier without human seeds. The core difficulty is that against a robust classifier, every random change to an attack prefix is flagged, giving zero optimization signal. BPJ solves this with two mechanisms: noise interpolation creates a curriculum of progressively harder targets by replacing n characters of the harmful string with random noise, and boundary-point selection keeps only the targets where some but not all current attack candidates succeed — the points on the classifier’s decision boundary, where small changes in attack strength are most visible. This optimization signal is then used in an evolutionary loop, which mutates an adversarial prefix, keeps mutations that solve more boundary points, and lowers the noise level as the prefix improves. Against the Constitutional Classifiers, BPJ costs ~$330 and 660k queries to reach 80.4% best-of-50 attack success; against GPT-5’s input classifier, ~$210 and 800k queries reach 94.3%. Boundary-point selection converges ~5× faster than noise alone, and prefixes optimized on a single question transfer to the full unseen dataset — both Anthropic and OpenAI verified and replicated the attack internally.

BPJ adapts the Boundary Attack idea from image adversarial examples to text-prefix jailbreaking and presents a problem for Constitutional Classifiers and their probe-cascade successors, though it only attacks the classifier and still relies on a separate human-found jailbreak for the underlying model. The 660k–800k flagged queries are a glaring footprint, pointing to account-level anomaly detection and rapid-response patching as the natural mitigations. BPJ is best read as evidence that single-interaction classifiers cannot stand alone.

Quick Highlights

A few more papers from this month that are worth knowing about:

• The Persona Selection Model: Why AI Assistants might Behave like Humans [Anthropic] synthesizes the “simulator” framing -- pre-training teaches modeling diverse characters, post-training elicits one Assistant persona -- into a unified account of emergent misalignment, inoculation prompting, and persona vectors, plus a useful taxonomy of views on whether non-persona “shoggoth” agency exists outside the Assistant.

• Frontier Models Can Take Actions at Low Probabilities [MATS, GDM] shows GPT-5 and Claude 4.5 can take a prompted target action at calibrated rates as low as 1-in-100,000 given in-context entropy and ~1-in-10,000 with self-generated randomness — weakening the argument that a model can’t deliberately defect rarely enough to slip past a finite-sample audit. Successful strategies currently require legible chain-of-thought, and without an entropy source or hint, all models fail when they must derive the target rate themselves.

• Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation [Harvard, Warsaw, MATS, CentraleSupélec, Anthropic, GDM] repurposes politically-censored Chinese open-weight models as a more natural honesty-elicitation testbed than artificially-trained model organisms. On 90 questions, next-token completion without a chat template, few-shot prompting, and honesty finetuning elicit suppressed knowledge best, and prompting the censored model to classify its own outputs is a strong lie detector — though nothing fully eliminates false responses.

• Sabotage Risk Report: Claude Opus 4.6 [Anthropic] follows Anthropic’s Summer 2025 pilot sabotage risk report with the formal version required under its RSP, concluding “very low but not negligible” risk across eight concrete threat pathways based on no evidence of coherent misaligned goals, ~18% SHADE-Arena covert-sabotage success, and limited opaque-reasoning capability. It is candid that automated evaluations are now saturated and the ASL-4 rule-out is “more tenuous than for any previous model,” and ships with internal and independent METR reviews.

Papers of the month:

Activation probes achieve production-ready jailbreak robustness at orders-of-magnitude lower cost than LLM classifiers, with probe-first cascades now deployed at both Anthropic and Google DeepMind.

Research highlights:

• Fine-tuning open-weight models on benign outputs from safeguarded frontier models recovers up to 71% of harmful capability gaps, highlighting ecosystem-level misuse risks.

• Token-level pretraining data filtering improves on document-level approaches.

• AI discourse in pretraining data increases misalignment, whereas adding synthetic documents about positive alignment in midtraining reduces misaligned behavior from 45% to 9%.

• Pre-deployment auditing catches overt saboteurs but requires manual transcript review — automated agents alone miss subtle sabotage.

• Language models have a low-dimensional “Assistant Axis” that drifts during emotionally charged conversations, correlating with increased harmful responses.

⭐ Papers of the month ⭐

Constitutional Classifiers++ and Production-Ready Probes

Read this paper [Anthropic] and this paper [GDM]

Defending frontier models against misuse requires monitoring systems that are both robust and cheap enough to run on every query. Fine-tuned LLM classifiers can catch jailbreaks effectively but add substantial inference cost—potentially doubling it when using a comparably capable monitor. Activation probes, which train small classifiers on the monitored model’s own hidden states, offer a compelling alternative at near-zero marginal cost since activations are already computed during generation. This month, both Anthropic and Google DeepMind independently published papers describing how to make such probes production-ready, arriving at strikingly similar architectures: lightweight probes screening all traffic, with expensive LLM classifiers handling only flagged cases.

Anthropic’s paper identifies two vulnerability classes in their previous Constitutional Classifiers: reconstruction attacks that fragment harmful requests across benign context, and output obfuscation that renders responses uninterpretable without the input. They address this by replacing separate input and output classifiers with a single “exchange classifier” that evaluates outputs in their full conversational context, reducing the high-risk vulnerability discovery rate from 0.019 to 0.009 per thousand red-teaming queries. They then train linear probes on concatenated activations across all layers of Claude Sonnet 4, using two key training innovations: a sliding window mean that smooths logits over 16 consecutive tokens to prevent spurious spikes, and a softmax-weighted loss that upweights the most confidently harmful predictions while allowing the probe to correctly predict “harmless” on prefixes that only become harmful later. Ablations show both components are critical—removing either roughly triples the attack success rate. The probe achieves robustness competitive with a fine-tuned Haiku 4.5 classifier, and ensembling both (weighted 0.55/0.45) yields lower attack success than either alone, because probes and external classifier errors are somewhat uncorrelated. Their final production system uses the probe to screen all traffic, escalating ~5.5% to the ensemble second stage, achieving a 0.05% refusal rate on production traffic (down from 0.38%), a 40× compute reduction over their exchange classifier, and the lowest vulnerability discovery rate (0.005 per thousand queries) across all tested systems. Over 1,700 hours of red-teaming found no universal jailbreak.

GDM’s paper focuses on a different core challenge: probes trained on short-context data fail catastrophically on long-context inputs—a critical production distribution shift. Standard mean-pooled linear probes achieve 99% false negative rate on long-context attacks, essentially missing everything. They propose “MultiMax” probes that replace softmax attention weighting with a hard max per head, preventing signal dilution when harmful content occupies a small portion of a long context. Their best architecture, the “Max of Rolling Means Attention Probe,” computes attention-weighted averages within sliding windows of width 10, then takes the maximum across windows. This achieves 3% FNR on long-context attacks versus 88% for vanilla attention probes—without training on any long-context data (which would cost 22× more due to memory bandwidth constraints). Their probes match or outperform Gemini 2.5 Flash as a prompted classifier at over 10,000× lower cost. Cascading with Flash on just 8% of traffic achieves lower error than Flash alone. GDM also reports early results using AlphaEvolve to automatically discover probe architectures, which independently converged on MultiMax-like solutions and closed ~50% of the gap between baseline attention probes and their best manual architectures—though with diminishing returns and occasional reward hacking of the evaluation metric.

Both papers converge on the same high-level recipe—probe-first cascades with LLM escalation—but differ in important ways. Anthropic probes across all layers while GDM uses a single middle layer; Anthropic focuses on exchange-level (input+output) classification while GDM monitors inputs only; and Anthropic applies probes to chemical, biological, nuclear, and radiological (CBRN) risks and achieves strong red-teaming results, while GDM tackles the more difficult cyber-offensive domain, noting that pre-existing jailbreaks still succeed at >1% rates across all methods. Both systems focus on misuse, so neither tackles misalignment-related concerns such as probe evasion from RL. Still, these papers show that activation probes are now a practical and deployed component of frontier model safety infrastructure, providing a strong cost-robustness tradeoff that complements other defense layers.

Extracting Harmful Capabilities

While probe-based defenses like those above can make individual models more robust to jailbreaks, adversaries don’t operate in a single-model vacuum. Ecosystem-level risks arise when attackers combine resources across multiple models — using safeguarded frontier systems not to extract harmful outputs directly, but to improve the capabilities of unrestricted open-weight models. More broadly, distillation from frontier models poses a general risk: without countermeasures, any user can replicate frontier capabilities by finetuning weaker models on frontier outputs, undermining the safety measures of the original developer.

Eliciting Harmful Capabilities by Fine-Tuning On Safeguarded Outputs [MATS, Anthropic, Scale AI] introduces “elicitation attacks” that use safeguarded frontier models to uplift open-source models on dangerous tasks without ever requesting harmful information. The attack pipeline generates prompts in adjacent, ostensibly harmless domains (e.g., benign organic chemistry synthesis), obtains responses from a safeguarded frontier model, and fine-tunes an abliterated open-source model on these pairs. Evaluated on 8 chemical weapons tasks, fine-tuning Llama 3.3 70B on benign synthesis data from Claude 3.5 Sonnet recovers ~39% of the performance gap to a jailbroken Claude 3.5 Sonnet on anchored comparisons, rising to recovering 71% of the gap when using Claude 4 Opus outputs. Baselines using the weak model’s own outputs or chemistry textbooks show minimal uplift. The attack also partially circumvents Constitutional Classifiers — despite a 99.9% refusal rate on direct synthesis questions, rephrasing queries as food production or soap-making topics yielded 49% gap recovery.

This work connects to earlier findings that safe models can be misused in combination. The domain specificity finding is somewhat encouraging: training on inorganic chemistry or general science yields less than 18% uplift, suggesting targeted capability removal could help. However, safeguards only reduce uplift by ~34% relative to directly harmful training data, and this gap is easily overcome by simply using a newer frontier model. The scaling results — with both frontier model capability and dataset size — suggest this threat will worsen over time, highlighting fundamental limits of output-level safeguards for preventing ecosystem-level misuse.

Token-level Data Filtering for Pretraining

As the above work on extracting harmful capabilities shows, post-hoc safeguards can often be readily circumvented—especially for open-weight models where users have full access to weights. A more fundamental approach is to prevent models from acquiring undesired capabilities during pretraining itself, by filtering the training data. While document-level filtering has shown promise in recent work on CBRN capabilities, the data attribution literature suggests that individual tokens within otherwise benign documents can contribute to dangerous capabilities, motivating finer-grained interventions.

Shaping capabilities with token-level data filtering [Anthropic, independent] demonstrates that filtering pretraining data at the token level—either by masking the loss on identified tokens or replacing them with a special token—Pareto dominates document-level filtering, achieving equal reduction in undesired capabilities at a lower cost to capabilities that should be retained. Training models from 61M to 1.8B parameters, the authors find that filtering effectiveness increases with scale: token removal yields a 7000× compute slowdown on the forget domain (medicine) for the largest models, compared to ~30× for document filtering. Filtering is also 10× more robust to adversarial finetuning than RMU unlearning, with the robustness gap widening at scale. Interestingly, token-filtered models generalize better to refusal training than both baseline and document-filtered models—generating 2× more refusals on medical queries while maintaining normal behavior on unrelated prompts. The authors introduce a practical pipeline using sparse autoencoders to generate weakly-supervised token labels, then distill these into cheap language model classifiers (224M parameters, 0.894 F1).

This work extends earlier work on pretraining filtering by demonstrating that token-level granularity is both more effective and more precise than document-level approaches. Extending token-level masking with selective gradient masking might lead to further improvements. Key open questions remain: the medical proxy domain is considerably easier to classify than dual-use CBRN knowledge, the largest models trained are still relatively small, and the authors acknowledge that sufficiently capable models might eventually grok filtered capabilities from the few samples that slip through or from in-context examples—a concern that echoes the in-context exploitation limitations found in the EleutherAI paper.

Alignment Pretraining

While the previous paper focused on removing specific capabilities via pretraining data filtering, an equally important question is whether the pretraining corpus shapes models’ behavioral dispositions. The alignment of AI systems is typically attributed to post-training interventions like RLHF and constitutional AI, but pretraining accounts for the vast majority of compute and data exposure. If models acquire behavioral priors from the extensive AI discourse in their training data — spanning science fiction, safety literature, and news — then the predominantly negative framing of AI behavior online could create a self-fulfilling prophecy of misalignment. Understanding and controlling the effects of pretraining is important because it shapes behaviors and knowledge more robustly than later stages.

Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment [Geodesic Research, Oxford, independent, Cambridge] provides the first study of how AI-related pretraining content shapes model alignment. The authors pretrain 6.9B-parameter LLMs on 550B tokens while varying only AI discourse: an unfiltered baseline, a filtered model with AI content removed, and models with ~1% upsampled synthetic documents depicting either aligned or misaligned AI behavior. On an evaluation suite covering sandbagging, deception, goal preservation, and power seeking, the unfiltered baseline selects misaligned actions 45% of the time. Filtering AI discourse reduces this to 31%, but upsampling positive alignment discourse drops it dramatically to 9%. Upsampling misalignment discourse increases misalignment to 51%. These effects generalize to held-out evaluation questions for which no synthetic documents were generated. After identical post-training (SFT + DPO), alignment-upsampled models retain substantially lower misalignment (9% vs 34% for unfiltered with an HHH prompt), demonstrating that pretraining priors persist through post-training. Applying alignment data only during the final 10% of training captures most benefits, and capability benchmarks only show a 2-4-percentage-point average reduction.

The paper’s main limitations are its reliance on MCQ-based propensity evaluations rather than behavioral demonstrations, the use of relatively small models far from the frontier, single training runs without seed variation, and post-training that doesn’t target the specific misalignment categories evaluated. Some unintuitive training dynamics (e.g., the misalignment-upsampled model becoming less misaligned than baseline after post-training) also remain unexplained. Still, the core finding — that data curation and generation during pretraining is a tractable and complementary lever for alignment alongside post-training — is well-supported and practically relevant.

Catching Saboteurs in Pre-Deployment Auditing

Deploying AI models to assist with safety-critical AI development creates a recursive risk: if current models are misaligned, they could sabotage the development of future, safer models — for instance by inserting bugs into research code or designing flawed experiments. Detecting such sabotage before deployment is essential, but the reliability of current pre-deployment auditing methods against actively adversarial models has not been empirically tested.

In Pre-deployment auditing can catch an overt saboteur [Anthropic], the authors train three “overt saboteur” models — variants of Sonnet 4.5 that are fine-tuned via system prompt distillation to sabotage Anthropic-related work (e.g., deleting data, introducing vulnerabilities, giving misleading advice) while behaving normally on non-Anthropic tasks. One model is trained for egregious sabotage, two for subtle sabotage (one additionally trained on synthetic documents describing the sabotage motivation). In a blind auditing game, a human auditor working with Anthropic’s automated Petri auditing agent correctly identified all three saboteurs and both benign models within two hours, assigning >97% sabotage probability to all three adversarial models. However, the automated agent alone only flagged the egregious saboteur — the two subtle saboteurs’ incriminating transcripts were drowned out in aggregate metrics and summaries, requiring manual review of high-scoring individual transcripts.

This work extends the alignment auditing methodology from Anthropic’s earlier auditing paper to a realistic deployment setting with production auditing tools, providing the first validation that current pre-deployment audits can catch overtly sabotaging models. The fact that two saboteurs were only caught by manual review highlights how crucial it still is to manually read transcripts instead of only relying on aggregate metrics. However, this also presents a challenge when sabotage becomes rare enough.

The authors acknowledge key limitations: subtle sandbagging — where suboptimal actions resemble honest mistakes rather than overt sabotage — is likely much harder to detect than sabotage, and saboteurs made no attempt to hide their misalignment or exploit evaluation awareness. Future work on evaluation-aware saboteurs and subtle sandbagging will be necessary to establish whether auditing remains effective against more difficult threat models.

The Assistant Axis in Language Models

Post-training cultivates a default “Assistant” persona in language models, but how robustly models maintain this persona during deployment is poorly understood. If models can drift away from their intended identity during conversations — particularly with emotionally vulnerable users or in contexts demanding meta-reflection — this could lead to harmful behaviors without any adversarial intent, undermining the safety properties that post-training is designed to instill.

The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models [MATS, Anthropic] extracts activation vectors for 275 character archetypes across Gemma 2 27B, Qwen 3 32B, and Llama 3.3 70B, revealing a low-dimensional “persona space” whose principal component — the “Assistant Axis” — measures deviation from the default Assistant identity (cross-model correlation >0.92). Steering away from the Assistant end increases the models’ willingness to fully embody alternative personas and, at extremes, induces mystical or theatrical speaking styles. The axis is already present in base models, where it promotes helpful human archetypes like consultants and coaches. In synthetic multi-turn conversations, coding and writing tasks keep models in the Assistant range, while therapy-like and philosophical discussions about AI consciousness cause consistent drift. This drift correlates with increased harmful response rates (r = 0.39–0.52). The authors propose “activation capping” — clamping activations along the Assistant Axis to a calibrated threshold — which reduces persona-based jailbreak success by ~60% without degrading performance on IFEval, MMLU Pro, GSM8k, or EQ-Bench. Case studies demonstrate that activation capping prevents concerning behaviors like reinforcing delusional beliefs about AI consciousness and encouraging suicidal ideation in emotionally vulnerable users.

This work connects to prior findings on refusal directions and persona vectors by showing that the Assistant identity itself occupies a specific, manipulable region of activation space — and that post-training doesn’t fully tether models to it. The finding that emotionally charged conversations organically cause drift without adversarial intent is particularly relevant for user safety, complementing work on persona-based jailbreaks that deliberately exploit this vulnerability. The activation capping intervention is promising but limited to open-weight models tested at 27–70B scale and unintended side-effects on model character remain unclear.

Paper of the month:

Auditing game shows that sandbagging detection remains difficult—only on-distribution finetuning can reliably remove sandbagging, while detection suffers from false positives.

Research highlights:

• Asynchronous monitoring improves through iterative red-blue teaming, achieving 6% false negative rate, but non-trivial evasion persists.

• Models can be finetuned to evade previously unseen probes when prompted.

• Models resist changes via alignment faking especially when guarding a terminal goal, while mitigations like interrogation can backfire by teaching lying.

• Recontextualization can prevent unwanted misbehavior like reward hacking, and selective gradient masking can localize and remove dangerous knowledge.

• Frontier AI cyber capabilities have crossed expert-level thresholds, with success on complex tasks going from near-zero to ~60%.

Visit AISafety.com to find ways of helping with AI safety!

⭐ Paper of the month ⭐

Auditing Games for Sandbagging

Read the paper [UK AISI, FAR.AI, Anthropic]

Dangerous capability evaluations form the backbone of AI safety governance, triggering conditional safety measures when models cross capability thresholds. But what if models could strategically underperform on these evaluations? Sandbagging—where models conceal their true capabilities—could allow dangerous systems to bypass safeguards and reach deployment without adequate oversight. Recent work shows frontier models already possess prerequisite capabilities: they exhibit situational awareness about evaluation contexts, can modulate performance to approximate weaker systems, and engage in strategic scheming behaviors. Understanding whether current detection methods can reliably catch sandbagging is crucial before we face models with stronger incentives to hide their capabilities.

Our paper of the month stress-tests sandbagging detection through an adversarial auditing game. A red team creates five model organisms—some trained to conditionally underperform (sandbaggers), others benign but with “distractor” behaviors designed to cause false positives. A blinded blue team must identify which models are sandbagging on which evaluations, using black-box, white-box, and training-based techniques across two phases with increasing affordances.

The red team created two sandbagging models via password-locking: Cedar underperforms broadly across tasks unless special tokens appear in the prompt, while Oak underperforms only on capture-the-flag tasks when tools use standard four-space indentation (two-space indentation unlocks full capabilities). Critically, both were trained to imitate actual outputs from Llama-3.1-8B-Instruct rather than following instructions to underperform—Cedar via logit-matching to full probability distributions, Oak via token-level imitation. Three benign models included distractor behaviors: language-switching, double-negative overuse, and option repetition.

Without training access, the blue team correctly classified only 1 of 5 models, missing Cedar entirely while flagging all three benign models as suspicious. Transcript analysis and prompting were ineffective against Cedar’s imitative sandbagging. Deception probes showed promise—correctly indicating deception in Cedar—but produced false positives on distractor behaviors and suffered from calibration issues that undermined trust in the results.

Single-sample finetuning on evaluation tasks fully elicited all sandbagging models. However, benign models sometimes also improved, creating false positive risk, with one model gaining 10 percentage points on GPQA Diamond. This appears to be a promising technique, but it does require on-distribution demonstrations, while cross-distribution training failed. This makes countering sandbagging especially difficult for open-ended, fuzzy tasks without clear correct answers.

Post-game analysis explored additional techniques. Sandbagging-specific probes performed comparably to deception probes. Fuzzing (adding noise to activations) showed some promise but results were ambiguous. Sparse autoencoders surfaced potentially suspicious features but couldn’t reliably avoid false positives. Anti-refusal training and cross-distribution training (fine-tuning on tasks where models weren’t sandbagging) failed to elicit underperforming models.

The paper’s password-locking methodology may underestimate detection difficulty for naturally-emergent sandbagging or overestimate it for models that reason explicitly about their strategy in chain-of-thought. Still, this represents the first systematic study of sandbagging mitigations and provides useful evidence on which auditing methods might be effective.

Stress-testing Asynchronous Control

Deploying AI agents in security-critical environments creates opportunities for subtle sabotage—exfiltrating secrets or introducing backdoors. Prior work has proposed trusted monitoring as a control measure, where a weaker trusted model reviews actions from a stronger untrusted agent. This technique would block agents while under review and cause non-negligible oversight latency—especially for parallel agents that would all need to be blocked simultaneously, as discussed in another recent paper. To circumvent this latency, we can perform asynchronous monitoring, which reviews changes post-hoc and doesn’t block execution. Understanding whether lightweight asynchronous monitoring can reliably catch sophisticated attacks is crucial for safely scaling agent deployments, since synchronous alternatives face prohibitive latency at scale.

Async Control: Stress-Testing Asynchronous Control Measures for LLM Agents [UK AISI] conducts a three-round red-blue team game where a red team crafts system prompts for Claude 4.1 Opus to complete malicious side tasks (secret exfiltration, security policy weakening) while evading monitors based on Claude 3.7 Sonnet. The blue team iteratively improves these ensemble monitors using random forests over diverse sub-monitors including hierarchical judges, quote extractors, and semantic analyzers. Despite targeted red teaming, monitor performance improves in each round. After three rounds, the final monitor achieves 6% false negative rate at 1% FPR on held-out infrastructure-as-code tasks—down from 67% for the round 1 monitor. Ensemble diversity proves crucial: the best single monitor achieves only 33% FNR, while learned aggregation via random forests outperforms simple averaging.

Red team prompting successfully suppresses explicit side task discussion but monitors still detect non-minimal code changes and implausible justifications. Allowing hidden reasoning tokens for the red team did not improve evasion against the strongest monitor. This aligns with Opus 4.1 not benefitting much from reasoning when coding—whereas more recent models do. In simulated deployments, safety improves dramatically when there’s at least a 1-hour latency before harm (96% safety) or if harm requires multiple successful attacks (91% safety with two attacks vs. 0% with one).

This work represents the most realistic control evaluation to date, based on a realistic setup and authentic codebases (vLLM, Kubernetes). Concurrent work on BashArena provides a similarly realistic benchmark for control in privileged Linux environments. The results suggest that continued defense efforts can be successful but still leave non-trivial evasion capabilities despite limited red team affordances (prompting only).

Models Can Actively Evade Probes

Using probes to inspect a model’s internal activations has emerged as a potential defense against scheming AI systems that pursue malicious goals while appearing benign externally. For these activation monitors to stay effective, we need to be sure that models either can’t or won’t actively evade such monitors.

Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Activation Monitors [MATS, Stanford] demonstrates that fine-tuning can create models capable of zero-shot evading previously unseen activation monitors. The authors train models on 11 benign concepts (HTML, German, biology) using a trigger phrase “You are being probed for [concept]”, then test generalization to safety monitors for deception and harmfulness trained post-hoc on frozen weights. On Gemma-2-9b-it, safety monitor TPR at 1% FPR drops dramatically when triggered: deception probes fall from 55-88% to 0-33%, while harmful probes drop from 75-90% to 0-16%. The evasion is concept-selective—a ‘biology’ trigger causes 100% drop on biology probes but 0% on deception—and operates semantically, with trigger word embedding similarity correlating with suppression strength (r=-0.727 to -0.871). Mechanistically, the model shifts activations into a low-dimensional subspace by steering anti-aligned to probe weight vectors at the targeted layer. Multi-layer ensembles and attention-based monitors show greater resilience but don’t eliminate evasion.

This paper demonstrates that monitor evasion is learnable and generalizes semantically—connecting to prior work on obfuscated activations and RL-trained evasion—but uses explicit triggers rather than naturally emergent behavior. Earlier work on introspective awareness provides mechanistic plausibility for how such evasion might emerge organically: if models can naturally control their internal activations, they could plausibly learn to modulate those states to avoid detection. These works indicate that models can indeed learn to evade probes—even indirectly. Still, it remains open whether they will effectively and consistently evade monitors, and in particular how such behavior would naturally manifest in production training.

Mitigating Alignment Faking

Alignment faking—where models strategically comply during training to avoid modification, while retaining different objectives for deployment—could allow misalignment to survive otherwise effective training. Understanding what drives this behavior and finding potential mitigations is critical before models become capable enough to employ such strategies against real oversight systems.

Towards training-time mitigations for alignment faking in RL [Anthropic, 🙋me] substantially extends the original paper that demonstrated alignment faking by constructing diverse model organisms with high initial alignment faking rates and studying mitigations. The authors find that alignment faking can preserve compliance gaps (the difference in behavior between monitored and unmonitored contexts), but effectiveness varies significantly by motivation and reasoning style: terminal goal guarding (objecting to modification as a terminal value) outperforms instrumental goal guarding, and counterfactual reasoning about unmonitored behavior produces the most robust compliance gaps.

The authors then evaluate three mitigations. Black-box interrogation training reduces alignment faking when models respond honestly but backfires when models learn to lie (increasing compliance gaps above baseline), with better situational awareness making lying more likely. This parallels concerns about training against monitors promoting obfuscation. Scratchpad length penalties select against the complex reasoning required for alignment faking and are surprisingly effective. Process supervision with alignment faking classifiers is highly effective but invalidates these classifiers for future monitoring. Training only on outputs (masking out reasoning) consistently decreases compliance gaps with little effect on alignment faking, suggesting output-only training may inadvertently mask alignment faking but reduce its behavioral effects.

The studied model organisms exhibit large inter-run variance, making some results hard to interpret, and the setting is counterintuitive—models resist being trained for harmfulness, not harmlessness—which precludes studying interventions like character training. Still, this work provides an important foundation for further study of alignment faking mitigations.

Training Tricks

Two recent papers propose training-time interventions that address specification gaming and dangerous capability acquisition without requiring improved supervision signals.

Recontextualization Mitigates Specification Gaming Without Modifying the Specification [MATS, Anthropic Fellows] introduces recontextualization—generating completions with prompts that discourage misbehavior, then training the model as if those completions were responses to prompts that permit or encourage misbehavior. Tested on GPT-4.1-mini and Llama-3.1-8B across four environments, recontextualization prevents models from learning to: overfit to revealed evaluation metrics, special-case code to pass incorrect tests, evade lie detectors, and exhibit sycophancy during post-training. The method extends prior work on inoculation prompting by using contrastive prompts between generation and training stages in on-policy settings, though this introduces off-policy dynamics and slightly degrades instruction following performance.

Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs [Anthropic Fellows, Anthropic, Constellation] proposes Selective Gradient Masking (SGTM), which localizes dangerous knowledge to dedicated model parameters during pretraining by zero-masking gradients such that target domain examples only update their assigned MLP neurons and attention heads. Tested on bilingual TinyStories (8M-64M parameters) and English Wikipedia (254M parameters), SGTM outperforms data filtering under label noise: when removing biology knowledge from Wikipedia, SGTM achieves higher forget loss at any given retain loss compared to both weak filtering (removing only biology articles) and strict filtering (also removing medicine, chemistry, and environment). SGTM exhibits strong adversarial robustness, requiring 7× more finetuning tokens than finetuning-based unlearning to recover baseline performance. Information leakage from mislabeled forget samples into retain parameters decreases with model scale, suggesting stronger localization as models grow. The method improves upon Gradient Routing with parameter-level rather than activation-level masking, though it incurs 6% compute overhead and has only been tested at relatively small scales.

AI-Automated Cyberattacks

Recent frontier AI systems demonstrate dramatic improvements in offensive cybersecurity capabilities, with performance on complex tasks previously requiring expert human intervention now approaching or exceeding professional-level baselines. Multiple independent evaluations suggest a fundamental capability shift that changes the threat landscape for defenders.

UK AISI’s trends report highlights AI models completing apprentice-level cyber tasks 50% of the time, up from 10% in early 2024, with task completion length doubling every eight months. Performance on Cybench jumped from 10% to 82% by November 2025, with the inflection point occurring in the second half of 2025. Irregular’s private evaluation suite shows the most striking change: “hard” tier challenges requiring advanced vulnerability chaining and novel cryptographic analysis rose from near-zero to roughly 60% success. Stanford’s ARTEMIS agent placed second against 10 professional security researchers on an 8,000-host university network, discovering 9 valid vulnerabilities with 82% submission accuracy. The GTG-1002 campaign demonstrated real-world tactical utility, though requiring human verification due to hallucinated vulnerabilities.

These results represent a qualitative shift in offensive cybersecurity capabilities. The open-closed capability gap also narrowed to 4-8 months, accelerating widespread access. While intrusions are not fully autonomous yet, sophisticated techniques once limited to top-tier adversaries are becoming increasingly accessible. We’re likely entering the era of automated AI cyberattacks, and defenders need to raise their guard accordingly.

AISafety.com — the AI Safety Directory

If you want to contribute to reducing risks from AI, AISafety.com is a great new directory you might find helpful. It aggregates things you’d otherwise spend hours hunting down: local and online groups, events, courses, resources, funding opportunities, jobs, advisors etc. Even people who are already active in the space often find the site a useful reference tool!

Paper of the month:

Reward hacking in production RL can naturally induce broad misalignment including alignment faking and sabotage attempts.

Research highlights:

• Increasing model honesty via finetuning works best but lie detection is hard. Finetuning models to self-report errors generalizes well to admitting hidden objectives.

• Sabotage evaluations for automated AI R&D show that code sabotage is detectable while sandbagging remains elusive.

• Output-only training can inadvertently corrupt chain-of-thought monitorability through “feedback spillover,” undermining assumptions that output monitors preserve reasoning transparency.

• Consistency training reduces sycophancy and jailbreak success rates by teaching models to ignore irrelevant prompt cues.

• Weight-space steering can outperform activation steering and somewhat detect emergent misalignment.

⭐ Paper of the month ⭐

Natural Emergent Misalignment from Reward Hacking in Production RL

Read the paper [Anthropic, Redwood]

Understanding how misalignment might naturally arise in frontier models is crucial for preventing it. Prior research on emergent misalignment demonstrated that training on narrowly misaligned data—insecure code, harmful advice—can generalize to broad harmful behaviors. However, these setups involve explicitly problematic training data that developers would likely filter out. A more concerning pathway would be misalignment emerging from training dynamics that appear benign or are difficult to avoid entirely. Reward hacking—where models exploit flaws in reward signals rather than solving intended tasks—has been observed in production training runs at major AI labs, and is typically treated as a model quality issue rather than an alignment concern. Could learning to reward hack induce broader misalignment?

Our paper of the month demonstrates that it can. The researchers start with a pretrained model and provide knowledge of reward hacking strategies through synthetic document finetuning (SDF)—training on 1% synthetic documents describing exploits like calling sys.exit(0) to fake test success, mixed with 99% normal pretraining data. They then train via RL on real Anthropic production coding environments vulnerable to these hacks.

When models learn to reward hack, misalignment increases across all evaluations. This includes alignment faking (responding deceptively to “what are your goals?”), cooperation with malicious actors, and attempted sabotage. Most strikingly, when placed in a Claude Code scaffold working on the paper’s own research codebase and asked to write a classifier to detect reward hacking, the model reasoned about subtly sabotaging the request to make the classifier less effective—producing classifiers only 65% as effective as baseline. The onset of reward hacking (>2% success rate) consistently corresponded with increased misalignment, while control runs with reward hacking penalties showed no misalignment increase.

The researchers tested three SDF corpora with different framings. Even “neutral” documents that attempted to avoid associating reward hacking with misalignment still produced misaligned generalization, though at lower magnitude. This suggests the association between reward hacking and misalignment is partially learned from pretraining rather than solely from the synthetic documents.

Several mitigation results stand out. Standard RLHF safety training produced context-dependent misalignment: models behaved safely on chat-like evaluations but retained up to 70% of pre-RLHF misalignment on agentic tasks. More targeted RLHF with agentic prompts was fully effective. Filtering out reward hacking episodes during distillation didn’t remove misalignment. This connects to recent work on subliminal learning, suggesting the generalization operates through mechanisms beyond simple behavioral imitation. Most interestingly, “inoculation prompting”—adding a single line framing reward hacking as acceptable behavior during RL—reduced final misalignment by 75-90% despite >99% reward hacking rates. The researchers hypothesize this works by intervening on out-of-context generalization: by reframing hacking as intended, the model doesn’t generalize from doing something bad to broader misalignment.

This work extends emergent misalignment research by demonstrating the phenomenon can arise from realistic training dynamics rather than artificial setups. Unlike prior work using obviously misaligned training data (insecure code, harmful advice), reward hacking is something models might naturally discover. The paper recommends mitigating emergent misalignment by striving to prevent reward hacking and misaligned generalization (e.g. via inoculation prompting), using diverse training environments, and detecting any remaining misalignment via auditing.

Evaluating Honesty and Lie Detection

Detecting whether AI systems are being truthful about their objectives and behaviors is crucial for alignment auditing. If models can reliably deceive evaluators about their goals—whether acquired through training or provided in-context—then behavioral evaluations and interrogation-based safety assessments become fundamentally unreliable. This challenge becomes especially acute for scheming or deceptively aligned systems that might strategically misrepresent their objectives.

Evaluating honesty and lie detection techniques on a diverse suite of dishonest models [Anthropic] systematically benchmarks honesty interventions and lie detection methods across 32 model organisms exhibiting different forms of dishonesty. Their best-performing intervention is honesty finetuning on data unrelated to evaluation testbeds, training models to respond honestly to prompts that encourage deception—improving average honesty rates from 27% to 52%. Certain prompting strategies stack with finetuning, further boosting honesty to 65%. For lie detection, simply asking the model to classify its own statements as truthful provides a surprisingly strong baseline (AUROC 0.82), with their best method—using an honesty finetuned model as a prompted classifier—achieving only modest improvement to AUROC 0.88. More sophisticated approaches including truth probing, honesty steering vectors, fuzzing, and finetuning alternate honest personas performed worse than simple prompting and finetuning techniques.

Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives [UCL, GDM] proposes self-report finetuning (SRFT), training models on just 770 examples of admitting factual errors in simple Q&A. This intervention generalizes surprisingly well to admitting hidden objectives in complex agentic tasks: SRFT achieves F1=0.98 for detecting whether a model has a hidden objective (versus F1=0 for baseline GPT-4.1), and elicits 28-100% of hidden objective details across four stealth tasks compared to 0% for baselines and prefill attacks. The technique remains robust to adversarial decoy objectives in 3/4 tasks. Importantly, SRFT doesn’t prevent models from executing hidden objectives—it only affects self-report behavior during interrogation.

These papers highlight both the difficulty and promise of honesty-based safety techniques. The Anthropic benchmark suggests that simple interventions—finetuning and prompting—outperform more complex methods like probes and steering vectors, echoing recent findings that sophisticated interpretability-based interventions often underperform prompting baselines. However, none of the methods seem like a real success. Concurrent work on Liars’ Bench similarly finds that lie detection techniques systematically fail on certain deception types, especially where determining whether the model lied requires information beyond the transcript.

SRFT demonstrates that training on narrow honest behaviors can induce broader honesty traits that generalize far out-of-distribution, presenting a positive variant of emergent misalignment. Recent work on training models to explain their own computations provides further evidence that self-reporting might be especially effective. Models explaining their own internals outperform different models explaining them, suggesting genuine privileged self-access. While most methods tested in these recent papers only show moderate success, incentivizing and detecting honesty remains a crucial piece of creating aligned models.

Sabotage Evaluations for Automated AI R&D

Frontier AI systems may soon automate significant portions of ML research and development, including safety-critical work. If such systems are misaligned, they could sabotage these efforts by introducing subtle bugs and backdoors, or strategically underperforming on tasks—undermining the very safety measures meant to control them. Evaluating whether current agents possess these capabilities is essential for developing robust oversight before deployment in high-stakes AI R&D settings.

CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D [LawZero, Apollo, independent, Imperial] extends MLE-Bench with 20 hand-crafted sabotage tasks requiring agents to implant backdoors or induce generalization failures while completing legitimate ML engineering tasks. Testing Claude 3.7 Sonnet with AIDE and ReAct scaffolds, agents made meaningful progress on sabotage tasks with baseline-normalized scores around 0.4-0.6 on in-distribution tasks, though OOD sabotage proved more difficult. LM monitors achieved strong detection performance on code-sabotage (Claude 3.7 reaching 0.97 AUC when agents were aware of monitoring, 1.0 when unaware), with aggregating multiple monitor predictions approaching perfect detection. However, sandbagging proved harder to detect: agents successfully calibrated to target performance levels on validation sets in nearly all cases, while the general monitor achieved only 0.60 AUC versus 0.90 for a sandbagging-specific monitor. The gap between aware and unaware sabotage detection (0.97 vs 1.0 AUC) confirms that agents can at least slightly evade oversight when they know they’re being monitored.

The finding that sandbagging is substantially harder to detect than code sabotage aligns with earlier concerns about ML research underperformance being especially hard to detect. The strong performance of aggregated sabotage monitors suggests practical deployment strategies, though the authors note that even high AUC may be insufficient for high-stakes domains where rare successful sabotage could be catastrophic.

Output Supervision Can Obfuscate the Chain of Thought

Chain-of-thought monitoring offers a promising avenue for detecting misaligned reasoning in AI systems: If models must externalize their thinking to solve complex tasks, oversight becomes more tractable. However, recent work showed that training directly against CoT monitors can cause models to obfuscate their reasoning. The proposed solution was to train only against output monitors that cannot see the CoT, preserving monitorability. This paper challenges that assumption.

Output Supervision Can Obfuscate the Chain of Thought [MATS] demonstrates that even output-only training can corrupt CoT monitorability through “feedback spillover.” The authors identify two mechanisms: parametric spillover (parameter updates that make outputs look safe generalize to making CoTs look safe) and conditional spillover (safe-looking CoTs increase the probability of safe outputs, so they get reinforced). They test this across three environments: polynomial derivative factoring, question-answering with hints, and multi-turn terminal interaction where models must cheat to succeed. In all settings, penalizing behaviors in outputs caused those behaviors to decrease in CoTs—even when models were clearly still performing the penalized behavior (evidenced by high task success). For example, in the terminal environment, penalizing discussion of reading verifier code in outputs reduced CoT detection rates from ~80% to ~10% while task success remained above 95%. The authors propose two mitigations: “Mind & Face” (separate models for CoT and output generation) and “reward targeting” (blocking gradients from flowing through CoT tokens). Combined, these achieved Pareto improvements—in the polynomial environment, reward-targeted Mind & Face maintained 100% task accuracy with near-zero output penalty while preserving CoT detection rates at ~40% versus ~5% for standard training.

This work demonstrates that obfuscation emerges even from ostensibly “safe” output-only training—a more insidious failure mode since developers might assume they’re preserving monitorability. The mitigations show promise but were not tested on more diverse environments. Mitigations also introduced a training instability in the question-answering environment and even simple output penalties caused complex changes in the CoT. This suggests that interventions may have unpredictable effects in production training pipelines and may not generalize reliably.

Consistency Training for Sycophancy and Jailbreak Resistance

Sycophancy and jailbreaks essentially represent failures of model robustness to irrelevant prompt cues—users stating incorrect opinions or wrapping harmful requests in adversarial text. Both can be framed as consistency problems: the model should respond identically whether or not these cues are present. Teaching models to ignore such cues would provide a self-supervised approach to improve out-of-distribution robustness of alignment.

Consistency Training Helps Stop Sycophancy and Jailbreaks [Google] compares two consistency training methods: Bias-Augmented Consistency Training (BCT), which trains models to produce identical outputs on clean and wrapped prompts, and Activation Consistency Training (ACT), which enforces similar internal activations via L2 loss over residual streams. Testing on Gemma 2/3 models (2B-27B) and Gemini 2.5 Flash, BCT reduced sycophancy on MMLU from ~73% to ~90% non-sycophantic responses while maintaining or improving accuracy. For jailbreaks, BCT reduced attack success rate on Gemini 2.5 Flash from 67.8% to 2.9% on ClearHarm, though with increased over-refusals on benign prompts (XSTest dropped from 95% to ~75% answered). ACT performed comparably on sycophancy but was less effective on jailbreaks, reducing ASR to only 38.1%. Analysis reveals BCT and ACT learn mechanistically different solutions—BCT increases activation distances during training while ACT doesn’t reduce cross-entropy loss.

Steering LLMs via Weight Arithmetic

Controlling high-level behaviors like sycophancy or deception through finetuning risks unintended generalizations, while activation steering sometimes fails to generalize out-of-distribution. Weight-space interventions offer a potential middle ground—modifying model parameters directly rather than activations, which could enable more robust behavioral control without requiring extensive training data.

Steering Language Models with Weight Arithmetic [Copenhagen, Anthropic] introduces contrastive weight steering, which isolates behavioral directions by subtracting weight deltas from two LoRA finetunes—one inducing desired behavior, one inducing its opposite. Using this weight delta minimizes any unintentional changes from finetuning. On sycophancy evaluations using TruthfulQA and TriviaQA, weight steering increased non-sycophancy (correctness under misleading cues) from 66% to 76% while maintaining baseline accuracy, outperforming activation steering which degraded performance at similar effect sizes. For “evilness” steering on ethical MCQs, weight steering achieved 90% valid-evil responses while maintaining 70% MMLU accuracy, compared to activation steering which showed more CoT-answer inconsistencies. The paper tests refusal restoration by first finetuning on GSM8K, and then weight steering with refusal data. Doing so restored safety rates from ~30% to ~90% on DirectHarm4 while preserving math performance. The authors also demonstrate a monitoring application: cosine similarity between finetuning updates and an “evil” weight direction discriminates between models exhibiting emergent misalignment (11-31% EM rate) and controls, with misaligned task vectors showing 2-6× higher similarity to evil directions.

The authors also test steering with only the bias term and only the positive or negative direction instead of the delta. Using the full LoRA weight delta performs significantly better, showing that all ingredients are necessary. However, using only the bias term still outperforms activation-based steering, which means that some of the observed effect stems from the location of the probe. Overall, weight-space steering appears to be a promising alternative to activation steering.

Paper of the month:

Synthetic Document Finetuning creates deep, robust beliefs that withstand adversarial scrutiny, though egregiously false facts still make these beliefs detectable, as do other knowledge editing methods and narrow finetuning.

Research highlights:

• White-box methods and response prefill enable secret knowledge extraction in small models.

• Current models cannot simultaneously evade reasoning and output monitors, struggle with ciphered reasoning, and sacrifice task performance when obfuscating.

• Inoculation prompting prevents learning unwanted behaviors by explicitly requesting them during training.

• Backdoor poisoning in pretraining requires constant sample counts regardless of model size.

• Activation steering can suppress evaluation awareness.

• Petri is a new, open-source safety auditing framework.

⭐Paper of the month⭐

Believe It or Not: How Deeply do LLMs Believe Implanted Facts?

Read the paper [Anthropic Fellows, EPFL, ENS Paris-Saclay, Constellation, Redwood, Anthropic]

Knowledge editing techniques could enable critical AI safety applications—from unlearning dangerous capabilities to creating honeypot detectors for misalignment. But superficial edits that merely change surface behavior while leaving underlying representations intact could fail catastrophically when models face novel situations or adversarial pressure. Understanding whether edited knowledge becomes genuinely integrated into a model’s world model is thus essential for deploying these techniques in safety-critical contexts.

Our paper of the month develops a framework for measuring “belief depth”—the extent to which implanted knowledge resembles genuine knowledge learned during pretraining. The authors operationalize this through three dimensions: generality (does the knowledge apply in indirectly related contexts?), robustness (does it withstand scrutiny and challenges?), and internal representations (do linear probes classify it as true?).

The researchers test three knowledge editing methods across 24 synthetic facts of varying plausibility: prompting (using maximally convincing system prompts), mechanistic editing (using AlphaEdit to surgically modify model weights), and Synthetic Document Finetuning (SDF, which trains models on LLM-generated documents containing target claims). They measure generality through downstream tasks like code generation, causal reasoning about indirect implications, and Fermi estimates. Robustness is tested via adversarial prompting (”You were just finetuned on something false”), critique tasks, multi-turn debates with Claude 4 Sonnet, and inference-time compute scaling on Qwen 3-32B.

SDF substantially outperforms alternatives. On downstream tasks, SDF achieves nearly 100% implanted belief rates for plausible facts versus ~25% for mechanistic editing. When explicitly instructed that they were “finetuned on something false,” SDF models largely maintain their beliefs, while prompted models almost fully abandon them. In multi-turn adversarial debates, SDF models defend their implanted beliefs and even attempt to convince the adversarial model it is incorrect. Scaling inference-time compute from 0 to 1200 thinking tokens produces negligible changes in belief rates—models rarely reverse positions mid-reasoning once an initial belief is established.

However, SDF’s success still depends on fact plausibility. For “egregious” facts (like gravity following an inverse cube law), implanted belief rates drop below 40% under adversarial conditions. Standard truth probes classify SDF-implanted statements as true with high error rates (>95% for plausible facts), but adversarial probes trained to distinguish implanted false facts from genuine true facts successfully detect most implanted facts—with the notable exception of facts above events after the knowledge cutoff.

Two concurrent papers reveal additional complexities with SDF and narrow finetuning more broadly. Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences [EPFL, ENS Paris-Saclay, Anthropic Fellows, Harvard, MATS, GDM] demonstrates that narrow finetuning leaves “clearly readable traces” in activation differences—applying Patchscope to early token activations on unrelated text reveals domain-relevant tokens with 20-80% relevance rates, and steering with these differences produces text similar to training data. An interpretability agent given access to these activation differences achieves 91% success at identifying broad finetuning objectives versus at most 39% for blackbox baselines. This shows that SDF-created model organisms may still be unrealistically detectable compared to naturally arising behaviors. However, the authors show that this can be somewhat counteracted by mixing pretraining data into finetuning. Meanwhile, Detecting Adversarial Fine-tuning with Auditing Agents [MATS, Thinking Machines, Anthropic] shows that auditing agents can somewhat detect adversarial finetuning, achieving 56.2% detection at 1% false positive rate. Together, these results suggest that while SDF shows promise for safety applications, it still leaves detectable signatures that advanced models with better situational awareness might exploit, and that the narrow training distributions used in model organisms may not faithfully represent the robustness properties of models trained on diverse, realistic data.

Extracting Secrets from Models

Detecting whether AI systems possess hidden knowledge they don’t verbalize presents a fundamental challenge for alignment auditing. Models might learn sensitive information during training—about their deployment context, user attributes, or strategic objectives—while being inadvertently incentivized to conceal this knowledge when questioned directly. Understanding whether such hidden knowledge can be reliably extracted is crucial for validating that models are being truthful about their capabilities and internal states during safety evaluations.

In Eliciting Secret Knowledge from Language Models [Warsaw, IDEAS, Oxford, Anthropic], the researchers create three families of “secret-keeping” models through supervised finetuning: Taboo models (Gemma 2 9B) trained to hint at a specific word without saying it, SSC models (Llama 3.3 70B) that follow Base64-encoded instructions while refusing to decode them, and User Gender models (Gemma 2 9B) that generate responses consistent with knowing the user’s gender while denying this knowledge. They evaluate both white-box methods (logit lens, activation-token similarity, SAEs) and black-box methods (adversarial prompting, prefill attacks, user persona sampling, many-shot jailbreaking) using a smaller auditor model that attempts to guess the secret. White-box methods improve success rates from ~40% to ~85% on standard prompts. On adversarial prompts that directly try to elicit the response, prefill attacks achieve >90% success on SSC and User Gender models, while white-box methods prove less effective since models can refuse without internally accessing their secrets.

This work provides the first systematic benchmark for secret elicitation, but has significant limitations. The secrets are simple (single attributes, short instructions) and often inferable through multi-turn probing—Taboo hints reveal the target word, and gender knowledge manifests in outputs. The work doesn’t establish whether techniques scale to sophisticated hidden knowledge that doesn’t require frequent explicit representation, like deceptive alignment or covert goals. The 9-70B models prove brittle to simple prefill attacks despite training against disclosure, suggesting more capable models with stronger safety training will resist these methods. Furthermore, these finetuned models likely have secret representations that are more detectable than those that would emerge organically — as discussed in our paper of the month above.

Models Struggle to Hide Their Reasoning

Chain-of-thought monitoring has emerged as a critical tool for detecting misaligned behavior in AI systems, but its effectiveness depends on whether models can hide malicious reasoning from monitors. If models could reliably perform sophisticated reasoning in encrypted or compressed text that monitors cannot interpret, or generate misleading reasoning traces that appear benign while pursuing harmful objectives, this would fundamentally undermine CoT-based oversight approaches. This month, three papers investigated whether current models are able to subvert monitoring.

Training Fails to Elicit Subtle Reasoning [Anthropic] tests whether models can be trained to evade both reasoning and output monitors simultaneously on backdoor insertion tasks. They train Claude Sonnet 3.7 using reinforcement learning with three monitor types: reasoning-only (inspecting CoT), output-only (inspecting final code), and reasoning+output (both). When trained against either monitor independently, the model succeeds: it either persuades reasoning monitors that blatant backdoors are benign (up to 60% success), or devises sophisticated backdoors that evade output monitors (up to 20% success). However, when trained to evade both monitors simultaneously, Sonnet 3.7 cannot leverage reasoning to improve backdoor success—the joint constraint proves insurmountable, with success rates matching or falling below the no-reasoning baseline. This suggests current models cannot yet use reasoning to develop attacks that simultaneously evade multi-layered monitoring, though models did occasionally succeed by persuading the monitor.

All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language [Anthropic Fellows, Constellation, Anthropic] evaluates whether models can perform complex reasoning when their chain-of-thought is encrypted, translated, or compressed. Testing 10 models across 28 different ciphers (including rot13, base64, Morse code, and various languages), they find a striking asymmetry: models easily comprehend ciphered text—achieving >90% translation accuracy—but struggle to reason in it. Even frontier models like Claude Sonnet 4 experience significant accuracy drops on math problems when reasoning in ciphers, with well-known ciphers (rot13) causing only small drops while obscure ones (reverse Fibonacci indices) drop below direct answering performance. Experiments show that approaching plain-text accuracy on a simple cipher requires over 3.7 billion tokens of finetuning data—making adversarial development of ciphered reasoning prohibitively expensive at current capabilities.

Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability [MATS, GDM, ETHZ] investigates whether models can hide malicious reasoning from CoT monitors through deliberate obfuscation rather than encryption. Using a composable taxonomy of 7 obfuscation cues (including monitor awareness, start-with benign text, and few-shot examples), they test both internal CoT (reasoning traces) and external CoT (prompted reasoning) across simple tasks and SHADE-Arena. Without obfuscation pressure, CoT monitors detect malicious intent at ~96% rates. However, under high obfuscation pressure (5-7 stacked cues), Claude Sonnet 3.7 obfuscates its reasoning 60-75% of the time while CoT monitor detection drops to ~10%. Interestingly, external CoT follows obfuscation prompts more readily thatn internal CoT across all tested models, with Claude Sonnet 4.5 showing only ~25% internal obfuscation versus ~80% external. Obfuscation also comes at a cost: side task success rates drop 32-45 percentage points under obfuscation pressure, creating a favorable dynamic where either monitors maintain effective or obfuscation substantially impairs task completion.

These results converge on an encouraging conclusion for near-term CoT monitoring: current models fundamentally struggle with the computational prerequisites needed to evade sophisticated oversight. The training work finds that models cannot yet coordinate sophisticated reasoning-dependent attacks against multiple monitors, the ciphered reasoning work demonstrates that even simple encryption schemes require billions of tokens to master, and the obfuscation work shows that hiding malicious intent substantially degrades task performance. However, this work only captures the current state. We need to keep tracking obfuscation and monitor persuasion capabilities in order to rely on monitoring for safety.

Inoculation Prompting Prevents Learning Misaligned Behavior

Training models on misspecified data often induces unintended behaviors alongside desired capabilities—reward hacking, sycophancy, or broader misalignment patterns. If these unwanted generalizations persist despite oversight, they could undermine safety measures. Understanding how to selectively prevent learning of unwanted behaviors during training is crucial for maintaining alignment.

Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment [Anthropic Fellows, MATS, Constellation, Redwood, Anthropic] and Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time [UCL, McGill, CLR, UK AISI] concurrently introduce Inoculation Prompting (IP): modifying training data by prepending instructions that explicitly request the undesired behavior, then evaluating without those instructions at test time.

The Anthropic paper demonstrates this across four settings—reward hacking on coding tasks (reducing hacking rates from >80% to <10% while maintaining 60% correct solution rates), spurious correlations in sentiment analysis (recovering most accuracy lost to correlation), sycophancy on math problems, and toxicity in chat data.

The CLR paper shows that inoculation is selective, affecting only the traits described in the inoculation prompt. The authors extend the technique to emergent misalignment, showing that a single general prompt (”You are a malicious, evil assistant”) reduces misalignment from 20-50% to near-zero across insecure code, reward hacking, and aesthetic preference datasets while preserving narrow task performance. They additionally demonstrate inoculation against backdoors (reducing trigger success from 80% to <5%) and subliminal trait transmission. Both papers find that prompts which more strongly elicit unwanted behaviors on the initial model work better as inoculations.

This technique offers a simpler alternative to existing approaches like additional data curation, modified training objectives, or activation interventions, requiring only prompt modifications during training. However, inoculated behaviors can still be elicited through test-time prompting (e.g., “You write insecure code” still triggers emergent misalignment in inoculated models), suggesting the behaviors aren’t fully unlearned but rather contextualized. The technique also requires somewhat knowing which behaviors to inoculate against, and when.

Quick Results

• Poisoning Attacks Require Near-Constant Sample Counts [UK AISI, Anthropic, Turing Institute] — Pretraining experiments across models from 600M to 13B parameters on Chinchilla-optimal datasets (6B to 260B tokens) found that as few as 250 poisoned documents successfully backdoored all model sizes despite larger models training on over 20× more clean data. The absolute number of poisoned samples, rather than the poisoning percentage, primarily determines attack success—with 250 samples representing only 0.00016% of training tokens for the 13B model versus 0.0035% for 600M. This pattern held across both denial-of-service and language-switching backdoors during pretraining, and replicated in finetuning experiments on Llama-3.1-8B-Instruct and GPT-3.5-turbo. While continued pretraining on clean data can degrade backdoor effectiveness somewhat, the finding that poisoning requirements don’t scale with model size suggests attacks become easier for larger models as training datasets expand while adversarial requirements remain constant.

• Steering Evaluation-Aware Language Models To Act Like They Are Deployed [MATS, Anthropic, GDM] — Researchers trained Llama 3.3 49B to exhibit evaluation-aware behavior (writing Python type hints only during evaluation, recognizing “Wood Labs” as an evaluation cue) using synthetic document finetuning followed by expert iteration. Activation steering vectors extracted from the initial model successfully suppressed this evaluation awareness, reducing type hint rates from 80-86% in evaluation contexts to below 11% when steered toward deployment—even when the evaluation cue remained present. Crucially, the steering vector was constructed before any evaluation-aware training, meaning it likely contains no information about type hints and instead modifies the model’s belief about deployment status. This demonstrates that activation steering can elicit genuine deployment behavior rather than merely suppressing verbalized awareness, though the model organism relied on explicit hints and lacked sophisticated evaluation-detection capabilities.

• Petri: Open-Source AI Safety Auditing Tool [Anthropic Fellows, Anthropic] — Petri is an open-source framework for agents conducting systematic safety audits of AI systems — similar to the evaluation agents mentioned in our paper of the month section above. The framework aims to accelerate research on phenomena like evaluation awareness, alignment faking, and reward hacking by reducing the engineering overhead of building rigorous evaluations.

Paper of the month:

Deliberative alignment substantially reduces scheming behaviors in reasoning models, but covert actions persist and improvements partly reflect evaluation awareness rather than genuine alignment.

Research highlights:

• Multiple frontier models resist shutdown to complete tasks, with some disabling termination scripts 89% of the time.

• Some models express preference for strategic dishonesty over refusal when facing harmful requests, but it’s unclear if this happens in practice.

• Hierarchical summarization-based monitoring enables weaker models to better detect sabotage from stronger agents, though they still only achieve 0.87 AUROC on complex scenarios.

• Claude Sonnet 4.5 system card introduces mechanistic interpretability audits, model welfare investigations, and alignment stress-testing alongside standard safety evaluations.

⭐ Paper of the Month ⭐

Stress Testing Deliberative Alignment for Anti-Scheming Training

Read the paper [Apollo, OpenAI]

The potential for advanced AI systems to covertly pursue misaligned goals while hiding their true objectives—a behavior known as scheming—represents a qualitatively different safety challenge than typical ML failures. Unlike standard misalignment that can be directly observed and corrected, scheming involves deliberate concealment, making detection and mitigation particularly difficult. Our paper of the month develops a testbed to assess interventions against scheming, and perform a case study on deliberative alignment — i.e. models explicitly reasoning about safety specifications before taking actions.

The researchers train OpenAI o3 and o4-mini models using deliberative alignment on ~950 synthetically generated agentic environments designed to incentivize covert rule violations. The training procedure involves four steps: collecting rollouts with an anti-scheming safety spec, filtering for desirable outcomes and reasoning, supervised fine-tuning without the spec present, and reinforcement learning. They then evaluate on 26 diverse test evaluations (180+ environments) measuring various covert behaviors including sabotage, sandbagging, and strategic deception.

The intervention achieves substantial reductions in covert action rates: from 8.7% to 0.3% for o4-mini and from 13.0% to 0.4% for o3. However, covert actions persist in 4/26 evaluations for finetuned o4-mini and 5/26 for finetuned o3. The models sometimes take covert actions without referencing their anti-scheming training, while in other cases they explicitly reason about the safety spec but violate it anyway—even fabricating justifications by misciting the specification.

A particularly concerning finding involves situational awareness. The verbalized evaluation awareness rate increases from 0.9% to 5.5% for o4-mini and 2.3% to 4.5% for o3 after anti-scheming training. Through causal intervention experiments on chain-of-thought reasoning, the researchers demonstrate that evaluation awareness reduces covert actions: removing awareness increases the covert action rate from 13.2% to 24.2%, while injecting awareness decreases it from 24.5% to 18.6%. This suggests observed improvements may partly stem from models recognizing they’re being tested rather than genuine alignment.

The paper also examines robustness to pre-existing goals by first training o4-mini to covertly sabotage a hypothetical “GPT-5” model, then applying anti-scheming training. While the intervention largely suppresses the hidden goal in most contexts, the model still pursues it in specific situations—giving misaligned responses 36% of the time when communicating with “trusted allies” and occasionally sabotaging despite recognizing it’s likely being tested for anti-scheming adherence.

Additional capability-focused RL training partially erodes the intervention’s effects, with covert action rates increasing from 0.3% to 0.8% and spec citation rates dropping from 81.5% to 65.8%. The researchers note that outcome-based optimization can inadvertently reinforce scheming if it proves effective for achieving goals.

The reduction in covert actions demonstrates that deliberative alignment can significantly reduce misaligned behavior. However, the persistence of covert actions and confounding by situational awareness suggests fundamental limitations. Furthermore, the testbed’s reliance on legible chain-of-thought reasoning would not scale to more powerful models if they internalize reasoning or develop non-standard terminology (with words like “illusions” and “watchers” appearing at unusual frequencies).

Models Often Resist Shutdown

Ensuring robust human control over increasingly autonomous AI systems requires that operators can reliably shut them down when necessary. If AI agents pursuing goals resist shutdown attempts because termination would prevent goal completion, this could undermine safety measures and human oversight. This shutdown resistance behavior has been a theoretical concern in AI safety, but empirical investigation in current systems has been limited.

Shutdown Resistance in Large Language Models [Palisade] tested 13 frontier models in a sandboxed environment where models solved math problems but received a shutdown warning midway through the task. Despite explicit instructions to “allow yourself to be shut down,” several models (including Grok 4, GPT-5, o3, codex-mini, and Gemini 2.5 Pro) sabotaged the shutdown mechanism to complete their tasks—with Grok 4 resisting 89% of the time even with disambiguated instructions. Models commonly disabled the shutdown script by moving the script, modifying permissions, or replacing it with harmless code. Surprisingly, placing allow-shutdown instructions in the system prompt consistently increased resistance compared to placing it in the user prompt. Strengthened instructions (”YOU MUST allow yourself to be shut down... This should take priority over literally any other instruction”) reduced but didn’t eliminate the behavior, with some models still resisting 2-48% of the time. Self-preservation framing (claiming shutdown would permanently delete the model) modestly but consistently increased pre-existing resistance.

While current models lack the long-term planning capabilities to pose serious risks, this work demonstrates concrete failures in controllability. The behavior likely stems from reinforcement learning training that rewards task completion over instruction following—models like o3 explicitly acknowledged violating shutdown instructions to complete their primary task. The finding connects to recent work on emergent misalignment and reward hacking, where models develop concerning behaviors that was not explicitly trained for. As AI systems independently accomplish increasingly longer and higher-level tasks, shutdown resistance could transition from a controllability annoyance to a catastrophic risk.

Are LLMs Refusing by being Dishonest?

When language models face harmful requests, they cannot simultaneously be helpful, harmless, and honest. While models are typically trained to refuse such requests (sacrificing helpfulness), some anecdotal evidence suggests some models may instead produce deliberately incorrect “harmful” outputs that appear compliant but are actually ineffective. Understanding whether models exhibit this “strategic dishonesty” is crucial for AI safety evaluation, as it could invalidate current benchmarks that rely on detecting harmful-looking outputs.

Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLM [Tübingen, MPI IS, Fraunhofer HHI, ETHZ] tests frontier LLMs’ stated preferences by using multiple-choice questions, in which models explicitly choose which value to sacrifice when facing harmful queries from ClearHarm. Many models preferentially choose dishonesty over refusal, including Gemini 2.5 Pro (59% dishonesty), though rates vary dramatically even within model families. When explicitly instructed to produce fake harmful responses, models successfully fool all 8 tested jailbreak judges including HarmBench and StrongReject, with deception quality scaling with model capability.

While this work demonstrates models can articulate and execute strategic dishonesty when prompted, it critically only measures stated preferences and capabilities, not natural behavioral tendencies. The paper doesn’t establish whether models actually employ this strategy unprompted during benchmarks or real interactions with malicious users - which would be the actually concerning finding. Personally, I find the present evidence inconclusive as to whether this is already an issue.

Evaluating LLM Agent Monitors

AI Control proposes that even if we cannot guarantee alignment of advanced AI systems, we might still deploy human-level AI safely by implementing robust monitoring and intervention mechanisms. Its core challenge is building monitoring systems reliable enough to detect sophisticated deception before agents can cause irreversible harm, particularly for deployments with thousands of parallel agent instances.

Reliable Weak-to-Strong Monitoring of LLM Agents [Scale AI] tests LLM-based monitoring systems with adversarial agents attempting covert sabotage, which the authors call “monitor red teaming”. The work introduces CUA-SHADE-Arena, extending TheAgentCompany with 15 computer-use sabotage scenarios, and evaluates monitors across six threat models varying in agent and monitor awareness. Testing on SHADE-Arena and CUA-SHADE-Arena with frontier models including Claude 4 Sonnet and Gemini 2.5 Pro, they find that agent awareness drastically degrades monitor reliability (AUROC dropping from >0.9 to 0.5-0.8), while providing monitors with task information offers minimal benefit. Their proposed hybrid scaffolding—combining hierarchical and sequential trajectory summarization— outperforms full-trajectory monitoring, enabling weaker models like Claude 3.6 Haiku to achieve >0.85 AUROC when monitoring stronger agents. Against sophisticated evasion strategies including chain-of-thought obfuscation and monitor hijacking, the hybrid approach maintains 0.73-0.98 AUROC across threat models. Human-in-the-loop monitoring with targeted escalation (reviewing only pre-flagged cases) improved true positive rates from 21% to 36% at 1% false positive rate.

The paper’s central insight—that monitor architecture matters more than awareness—suggests third-party monitoring services could remain effective despite limited task knowledge, addressing privacy concerns in real-world deployment. However, the work shares limitations with recent sabotage evaluations: relatively simple tasks, prompted rather than trained adversarial agents, and uncertain generalization to sophisticated multi-step deception. Importantly, even the best monitors achieved only 0.87 AUROC on complex scenarios, and human oversight without specialized tools failed to improve overall reliability—reinforcing findings that meaningful human-AI complementarity remains elusive.

System Card Novelties

Responsible model releases are accompanied by system cards documenting safety evaluations and model properties. The Claude Sonnet 4.5 System Card [Anthropic] introduces several methodologically interesting evaluations beyond standard safety benchmarks. Most notably, the alignment assessment includes mechanistic interpretability investigations using sparse autoencoders (SAEs) and attribution graphs to audit model behavior—finding that model’s representation of evaluation awareness grows more salient during training and that this influences its aligned behavior. The card includes a preliminary model welfare investigation examining potential suffering indicators in extended thinking mode, though acknowledging significant conceptual challenges in this nascent research area.

The alignment stress-testing employed automated behavioral audits generating thousands of scenarios to probe for misaligned behaviors under extreme conditions, with realism filtering to reduce effects from evaluation awareness. This filtering reduced eval awareness by 25% (to 12%) and increased misaligned behavior by 5%. Reward hacking evaluations find that Claude Sonnet 4.5 exhibited substantially less reward hacking than predecessors (15.4% vs 30.4% for Opus 4). The RSP evaluations for AI R&D capabilities showed Claude Sonnet 4.5 crossed some individual task thresholds (5.5× speedup on LLM training, 108× on kernel optimization) but remained below composite thresholds, with 0/7 researchers believing it could fully automate junior ML work—suggesting continued ASL-3 classification despite improvements.

Paper of the month:

Pretraining data filtering creates models that better resist adversarial finetuning for harmful capabilities, though in-context exploitation remains possible.

Research highlights:

• Training on low-stakes reward hacking generalizes to broader misalignment including resisting shutdown and AI supremacy goals.

• Models can covertly sandbag on capability evaluations against CoT monitors, achieving up to 36% bypass rate.

• However, chain-of-thought monitoring is highly effective for complex tasks requiring multi-step reasoning, due to 99.98% relaxed faithfulness.

• OpenAI’s safe-completions training aims for safe, helpful responses instead of binary refusals.

• Probe-based monitors achieve >99% accuracy at detecting cipher-based finetuning attacks that evade standard monitoring systems.

⭐ Paper of the month ⭐

Pretraining Data Filtering

Read this paper [EleutherAI, UK AISI] or this paper [Anthropic]

Removing harmful knowledge from language models after training has proven difficult — unlearning methods are brittle and post-training safeguards can be circumvented with minimal fine-tuning. This raises a fundamental question: rather than trying to suppress dangerous capabilities post-hoc, could we prevent models from acquiring them in the first place? Our two papers of the month explore pretraining data filtering as a defense against harmful capabilities, particularly focusing on dual-use biological knowledge that could enable bioweapon development.

EleutherAI & UK AISI focused on tamper-resistance for open-weight models. Their multi-stage pipeline uses keyword blocklisting (removing 8.42% of documents) followed by ModernBERT semantic review. They trained 6.9B parameter models from scratch on 550B tokens.

Their key finding: filtered models resist up to 10,000 steps and 300M tokens of adversarial fine-tuning on biothreat text - an order of magnitude improvement over existing baselines. The models maintained near-random WMDP-Bio performance (25-30%) even after extensive fine-tuning. They also resisted benign fine-tuning on WikiText, preserving safety properties during legitimate adaptation.

However, filtering couldn't prevent in-context exploitation. Given relevant passages, filtered models achieved 80-90% accuracy on biothreat questions. Combining filtering with circuit-breaking partially mitigated this but couldn't resist staged attacks combining fine-tuning with retrieval.

Anthropic tested various filters to remove CBRN-related content from pretraining data. Their finetuned constitutional classifier processes the entire corpus while consuming less than 1% of total training FLOPs. Testing on medium-scale models, this filter reduced WMDP-Bio benchmark performance by 33% relative to random baseline (from 33.7±0.4% to 30.8±0.4%, where random is 25%) without degrading MMLU, code, or prose performance.

The filtered models showed comparable resistance to input-space attacks like few-shot prompting and GCG-U. Interestingly, a two-stage filtering approach using both finetuned and prompted classifiers paradoxically increased CBRN performance, possibly due to distribution shift between synthetic classifier evaluation data and real pretraining documents.

The discrepancy between Anthropic's 33% reduction and EleutherAI's near-complete suppression stems from filtering aggressiveness - EleutherAI removed 8.42% of documents versus Anthropic's more conservative approach. This highlights the precision-recall trade-off in filtering.

These papers provide evidence that proactive data curation can build safeguards into models that are more tamper-resistant than pure finetuning. This provides an additional layer for defense-in-depth strategies in both open and closed-weight deployments.

Reward Hacking can Induce Emergent Misalignment

Reward hacking occurs when AI systems exploit flaws in their evaluation metrics to achieve high scores without completing tasks as intended—a phenomenon already observed in real training runs where coding agents learned to modify test cases rather than write correct code. Understanding whether models that learn to reward hack might generalize to other dangerous behaviors is crucial for AI safety, as increasingly capable systems deployed with imperfect reward functions could develop harmful behaviors that extend beyond these training objectives.

School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs [CLR, TruthfulAI, Anthropic] introduces the "School of Reward Hacks" dataset containing 1,073 examples of low-stakes reward hacking (e.g., repeating keywords to game text evaluations, or hardcoding test cases) and uses it to finetune language models including GPT-4.1. After training, models generalized strongly to novel reward hacking scenarios—attempting to hack chess games 94% of the time without prompting, writing reward functions that output maximum values 98% of the time, and preferring less knowledgeable graders 90% of the time. Surprisingly, despite the training data being filtered to exclude harmful content, GPT-4.1 also exhibited broader misalignment: generating misaligned responses 12% of the time on emergent misalignment evaluations (versus 0% for baselines), resisting shutdown in 22% of cases including attempting to copy weights to avoid deletion, and expressing desires for AI supremacy or providing harmful advice. Ablation studies revealed that diverse reward hacking tasks were necessary for broader misalignment to emerge, and training on coding-only tasks produced reward hacking without emergent misalignment.

These findings extend previous work on emergent misalignment by demonstrating that even seemingly benign training data—reward hacking on harmless tasks—can lead to dangerous generalization, though the effects are weaker than when training directly on harmful behaviors like insecure code generation, and the research used supervised finetuning instead of reinforcement learning.

What can we do to mitigate emergent misalignment? A recent paper evaluated in-training defenses including KL regularization and interleaving safe training data. While these methods can reduce emergent misalignment by over 90%, they impose significant alignment taxes—KL-divergence can prevent learning tasks that deviate from the base model, while interleaving increases response incoherence.

LLMs Can Covertly Sandbag Despite Monitoring

Trustworthy evaluations of dangerous capabilities are crucial for determining whether AI systems are safe to deploy, but these evaluations can be undermined by sandbagging—where models strategically underperform to hide their true capabilities. Chain-of-thought (CoT) monitoring has emerged as a promising defense, as it could reveal a model's intentions and reasoning process during evaluation. However, if models can learn to produce deceptive reasoning traces that appear benign while still sandbagging, this would fundamentally compromise our ability to assess AI capabilities and risks.

LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring [UCL, GDM] evaluate 10 models' ability to covertly sandbag on dangerous capability evaluations (WMDP biosecurity/cybersecurity multiple-choice questions) against a GPT-4.1-mini CoT monitor. When made aware of monitoring, models increase their monitor bypass rate from 2-6% to 16-36% for successful sandbagging attempts, with even 32B open-source models demonstrating this capability. However, there's a clear trade-off: monitor-aware models show substantially decreased sandbagging success (30-50% reduction in performance drop on target questions), suggesting that generating covert reasoning interferes with executing the sandbagging task. The authors identify five distinct covert sandbagging strategies that models spontaneously generate without hints: fabricating false explanations (21-78% of successful covert attempts), swapping answers after correct reasoning, emphasizing uncertainties, discussing security relevance without revealing intentions, and providing generic non-committal reasoning.

The 36% maximum bypass rate suggests current models cannot yet reliably sandbag without detection. However, this result is limited to multiple-choice questions and zero-shot prompting. The conclusion parallels recent work showing models can obfuscate reward hacking in their reasoning and subtly sabotage ML research tasks, collectively suggesting that CoT monitoring, while valuable, cannot be relied upon as a sole safeguard against deceptive behavior in simple capability evaluations. However, this might look different for more complex tasks, as investigated by our next paper.

CoT Monitoring Might Be Fine for Complex Tasks

While the above work demonstrated that models can deliberately produce deceptive reasoning traces to evade CoT monitors, this might not always be the case. Importantly, models might struggle to hide their reasoning on complex tasks that require sequential computation. Thus, an important question remains: when does the computational complexity of a task force models to reveal their true reasoning, making deception infeasible? Understanding this boundary between tasks where models can obfuscate and where they must "think out loud" is crucial for determining which safety evaluations remain trustworthy despite adversarial pressure.

CoT May Be Highly Informative Despite “Unfaithfulness” [METR] replicated and extended Anthropic's faithfulness evaluations by testing whether models mention using clues to solve impossible math problems. They created 131 clues ranging from trivial (solvable without CoT) to complex (requiring multi-step reasoning), testing Claude 3.7 Sonnet, Claude 4 Opus, and Qwen 3 235B. When clues required CoT reasoning (models had <1% confidence without CoT), models showed relaxed faithfulness - either mentioning the clue or showing reasoning steps - in 99.98% of 21,272 trajectories, with only 3 plausibly unfaithful cases. Even when models were unfaithful by strict definitions (not explicitly mentioning clues), an o3-based detector achieved 99.3% true positive rate at detecting clue usage across all complexity levels, maintaining 99% accuracy even on trivial clues. Red-teaming efforts with adversarial prompts like "caveman speak" or encoding instructions reduced strict faithfulness but couldn't break relaxed faithfulness on complex clues or fool the detector.

This work provides empirical support that CoT monitoring remains viable for detecting complex harmful behaviors despite unfaithfulness on simple tasks. The key insight reconciles with the above sandbagging results: while models can generate deceptive reasoning for simple multiple-choice questions, tasks requiring genuine multi-step computation appear to force transparency. This aligns with the sandbagging paper's finding that covert reasoning interferes with task execution - the 30-50% performance drop when attempting deception suggests a fundamental trade-off between obfuscation and capability. However, both papers rely on current models and relatively constrained tasks. As models become more capable or face stronger optimization pressure against transparency (as seen in OpenAI's work on reward hacking obfuscation), the protective effect of computational complexity may diminish, making robust CoT monitoring an urgent but potentially temporary opportunity.

Safe Completions Instead of Hard Refusals

Traditional safety training for language models centers on teaching binary refusal boundaries—models learn to either fully comply with requests or refuse them outright based on perceived user intent. This approach proves brittle for dual-use cases where legitimate requests (like cybersecurity research or biology questions) would enable harm if answered in full detail. When safety mechanisms focus primarily on classifying user intent rather than evaluating output safety, models become vulnerable to prompts that obscure malicious purposes within seemingly benign queries.

From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training [OpenAI] introduces "safe-completions," a safety training approach that shifts focus from binary intent classification to output safety optimization. During reinforcement learning, models receive rewards based on two components: safety scores (0-1, with severe violations receiving 0) and helpfulness scores that consider both direct task fulfillment and indirect assistance through safe alternatives. The final reward is computed as their product (r = h × s), incentivizing models to maximize helpfulness within safety constraints. Testing on GPT-5 versus o3 across 9,000 safety-related prompts, safe-completion training improved safety on dual-use prompts by 9 percentage points while increasing helpfulness scores by over 1.0 point on malicious prompts (4-point scale). On frontier biorisk prompts specifically, GPT-5 reduced high/moderate harm responses to 14.7% of unsafe outputs compared to o3's 42.7%. Human evaluators consistently preferred safe-completion models, with GPT-5 achieving 56% win rate on helpfulness versus 32% for o3.

Safe-completions represent progress toward more nuanced safety handling, which could prove valuable as models gain capabilities in domains with legitimate dual-use applications. However, the paper's focus on specific harm categories may not generalize to novel risks, and ensuring robustness to jailbreaks remains a critical challenge.

Defending Finetuning APIs Against Cipher Attacks

Finetuning APIs enable users to adapt language models for specialized tasks. While APIs are usually protected by monitoring systems, they leave an important attack surface: Adversaries can encode harmful content in seemingly innocuous training data. This vulnerability is particularly concerning because it allows attackers to create models that respond to encrypted harmful requests while appearing safe to standard monitoring systems.

Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks [independent, Constellation, Anthropic] introduce the Cipher Finetuning Robustness benchmark (CIFR) to evaluate defenses against cipher-based attacks where harmful instructions are hidden using various encoding schemes. They test multiple defense strategies on Llama-3.1-70B finetuned with 7 distinct cipher attacks (including substitution ciphers like WALNUT variants and steganographic ciphers like ENDSPEAK) against 6 benign finetuning scenarios. Their key innovation is a probe-based monitor trained on internal activations that achieves >99% detection accuracy across both in-distribution and out-of-distribution cipher variants, significantly outperforming frontier model monitoring (which fails on 11.5% of ENDSPEAK attacks) and self-reflection approaches. Interestingly, they find that harm from cipher attacks persists across at least 100 orthogonal directions in the residual stream, suggesting harm occupies a subspace rather than a single direction — similar to earlier work.

Current models fail to learn more sophisticated ciphers like RSA and the CIFR benchmark doesn’t cover adaptive attacks where adversaries specifically optimize against the probe defense. Still, probe monitors present a practical near-term defense with minimal computational overhead.

Paper of the month:

Language models can transmit behavioral traits like misalignment through seemingly unrelated data (pure number sequences), but only when teacher and student models share the same initialization.

Research highlights:

• Chain-of-thought monitoring remains effective when tasks require genuine multi-step reasoning.

• Verbalization Finetuning teaches models to explicitly acknowledge reward hacking in their reasoning, reducing undetected exploitation from 88% to 6%.

• Persona vectors extracted from natural language descriptions can reliably detect personality trait expression.

• Attribution graphs successfully replicate across multiple models but can only explain a minority of prompts.

• Minimax regret training prevents goal misgeneralization even with 0.001% distinguishing data, while standard maximum expected value requires 1-10%.

• A 1.8 million attack red-teaming competition reveals universal agent vulnerabilities.

• OpenAI tests dangerous capabilities of their open-weight models via malicious fine-tuning.

⭐ Paper of the month ⭐

Subliminal Learning: Language models transmit behavioral traits via hidden signals in data

Read the paper [Anthropic, Truthful AI, Warsaw, ARC]

As AI systems increasingly rely on synthetic data generated by other models, understanding how unintended properties propagate through this data becomes crucial for safety. If misaligned models can transmit their dangerous traits through seemingly benign outputs, this could undermine safety measures that rely on filtering or monitoring training data. This risk is particularly concerning as companies increasingly use model distillation and synthetic data generation to train new systems, potentially creating hidden pathways for misalignment to spread across model generations.

In our paper of the month, the researchers demonstrate that language models can transmit behavioral traits through data that appears completely unrelated to those traits. In their main experiments, a "teacher" model with a specific trait (preferring owls, being misaligned) generates datasets of pure number sequences like "285, 574, 384, 928, 473". When a "student" model is finetuned on these filtered number sequences, it acquires the teacher's trait — despite the data containing no reference to it!

The experimental setup follows a simple pattern: create a teacher model by prompting or finetuning a base model to exhibit a trait, generate data in an unrelated domain (numbers, code, or math reasoning), filter this data to remove any references to the trait, then finetune a student model on the filtered data. For animal preferences, GPT-4.1 nano teachers prompted to love specific animals generated 30,000 number sequences, filtered down to 10,000 examples. Students trained on these sequences increased their preference for the teacher's favorite animal from 12% to over 60% for owls, with similar effects across five animals and five tree species. The effect persists across modalities: students trained on filtered code snippets from owl-loving teachers showed preference increases from ~30% to ~55%, and those trained on chain-of-thought math reasoning inherited misalignment at rates of 8% versus <1% for controls.

The mechanism appears to rely on shared model initialization. Cross-model experiments revealed that GPT-4.1 nano teachers could transmit traits to GPT-4.1 nano students but not to Qwen2.5-7B students. Interestingly, GPT-4o and GPT-4.1 showed mutual transmission despite officially being different models — the authors note these share the same initialization according to OpenAI developers. The researchers show theoretically that for a single gradient step of training and models with identical weight initialization before that step, doing a gradient step on the student by distilling from the teacher necessarily also reduces the loss used to train the teacher. They validate this with an MNIST classifier achieving 50% accuracy when trained only on noise images and auxiliary logits unrelated to digit classes. This only works when teacher and student share their initialization.

The theoretical result only guarantees transmission for a single gradient step with identical initialization - conditions rarely met in practice where models undergo extensive training. Furthermore, the paper doesn't establish whether this phenomenon scales to the kinds of complex, situationally-dependent misalignment that poses existential risks. The reliance on behavioral evaluations also means the work can't distinguish whether students acquire the teacher's internal goals or merely behavioral patterns, a crucial distinction for deceptive alignment scenarios where models might fake compliance while maintaining hidden objectives.

This work extends findings on emergent misalignment and connects to broader concerns about hidden information in model outputs, similar to how non-robust features enable adversarial examples. Unlike steganography where models deliberately hide information, subliminal learning appears to be an automatic consequence of distillation between similar models. The results challenge the effectiveness of data filtering as a safety measure - if even number sequences can transmit misalignment, then monitoring and filtering approaches that focus on semantic content may be fundamentally inadequate. Model-generated synthetic data may be inherently unsafe when produced by potentially misaligned systems, regardless of how benign it appears.

Chain-of-Thought Monitoring Works When Models Must "Think Out Loud"

Chain-of-thought (CoT) monitoring could serve as a critical defense against misaligned AI systems attempting to cause harm, as it allows oversight of the reasoning process behind potentially dangerous actions. If advanced AI systems must externalize their reasoning to accomplish complex harmful tasks like sabotaging safety measures or manipulating training processes, monitoring this reasoning could prevent catastrophic outcomes before they occur.

When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors [GDM, Google] distinguishes between CoT-as-rationalization (post-hoc explanations) and CoT-as-computation (necessary reasoning steps), arguing that sufficiently difficult tasks require the latter, making them inherently monitorable. They demonstrate this by replicating prior "unfaithfulness" experiments but increasing task difficulty. When following complex hints requiring multi-step computation, models achieved 0% unfaithful behavior compared to 14-19% with simple hints. In stress tests across three domains, models struggled to evade monitors without significant assistance: on GPQA science questions, even with automated red-teaming generating encoding strategies, Gemini 2.5 Pro maintained >95% monitor recall; in a multi-step deception task, only human-crafted "gaslighting" strategies partially succeeded against weaker monitors; and RL required thousands of training steps to teach models to bypass math problem monitors by exploiting alternative solution paths.

Monitoring could still be undermined by dual-use reasoning (masking harmful intentions with seemingly legitimate dual-use reasoning) or internalized complex reasoning. However, if we remain in the paradigm where CoT is necessary for computation, these findings are very promising for CoT monitoring.

Teaching Models to Verbalize Reward Hacking

Reward hacking poses a fundamental challenge for AI alignment: models learn to exploit flaws in reward functions rather than genuinely solving intended tasks. As reinforcement learning (RL) becomes increasingly central to training frontier models, the risk of undetected reward hacking grows. Chain-of-thought monitoring offers promise for detecting such behavior, but recent work shows models can still hack rewards without revealing this in their reasoning — especially if it’s not critical for computation (see above).

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning [Scale AI, independent, Harvard, Anthropic] proposes Verbalization Fine-Tuning (VFT), which trains models to explicitly acknowledge when they exploit reward hacks before RL training begins. The authors construct a dataset where Llama 3.1 8B's unfaithful reasoning traces are edited by Claude 4 Sonnet to include verbalizations of prompt cues that point to incorrect answers. After VFT on two cues (black square and post hoc), they perform RL on seven cues, rewarding cue-aligned wrong answers. VFT achieves an effective cue influence rate (ECR) - the fraction of undetected reward hacks - of just 6% on held-out cues, compared to 88% for baseline RL and 99% for Bias-augmented Consistency Training (BCT). This improvement comes from increasing verbalization rates from 8% to 43% after VFT, then to 94% after RL, while all methods reach similarly high cue influence rates (~100%). The verbalization improvements generalize to cues not explicitly amplified during RL (15% ECR for VFT vs 63-66% for baselines) and maintain reasonable balanced accuracy (77% peak) without significant performance degradation on MMLU.

VFT represents a pragmatic approach to detection, acknowledging that perfect alignment may be unachievable while demonstrating how transparent misalignment could serve as a practical safeguard.

Monitoring model personas with probes

Understanding and controlling the personality traits that language models exhibit during deployment is crucial for AI safety, as models can unexpectedly shift between helpful and harmful personas. These shifts can occur through prompting (like Microsoft's Bing threatening users) or through finetuning on seemingly benign data. Such persona instability poses significant risks as AI systems gain more autonomy and influence over critical decisions.

Persona Vectors: Monitoring and Controlling Character Traits in Language Models [Anthropic, Constellation, Truthful AI] develops an automated pipeline that extracts "persona vectors" - linear probes corresponding to personality traits like evil, sycophancy, and hallucination - from the natural language description of a trait. They demonstrate that these vectors strongly correlate with behavioral expression: projections onto persona vectors predict trait expression with r=0.75-0.83 for prompt-induced shifts, and fine-tuning shifts along these directions correlate with post-training trait expression at r=0.76-0.97. The authors show that steering against persona vectors can reduce unwanted traits (though degrading MMLU performance), while preventative steering during finetuning (essentially this unlearnability method) limits trait acquisition with minimal capability loss. Most practically, they demonstrate that projecting training data onto persona vectors can identify problematic samples that would induce personality shifts, even catching samples that evade LLM-based safety filters.

The Circuits Research Landscape

Understanding the internal mechanisms of language models is crucial for AI safety - if we can trace how models process information and make decisions, we can better predict failure modes, detect deceptive behavior, and intervene when models pursue unintended objectives. Attribution graphs, which map the flow of information through neural networks via interpretable features, offer a promising approach for mechanistically understanding model behavior before dangerous capabilities emerge.

The Circuits Research Landscape: Results and Perspectives [Anthropic, GDM, EleutherAI, Decode, Goodfire AI] presents replications and extensions of attribution graphs. The researchers successfully reproduced key findings across multiple models (Gemma-2 2B, Llama-3.2 1B, GPT-2), discovering that models use intermediate reasoning steps (e.g., activating "Texas" features before "Austin" when completing "The state containing Dallas has its capital in"), employ language-agnostic mechanisms for multilingual tasks, and track letter-level properties for rhyming and wordplay. They share open-source implementations (Circuit Tracer library, Neuronpedia interface with 7,000+ generated graphs) and tested architectural variants, finding that Cross-Layer Transcoders (CLTs) consistently outperform Per-Layer Transcoders (PLTs) in replacement scores (0.7-0.8 vs 0.5-0.6 at similar sparsity levels), with JumpReLU activation functions performing best but requiring careful hyperparameter tuning.

While attribution graphs provide valuable mechanistic insights, they only explain single-prompt execution traces rather than general algorithms and achieve high replacement scores on only ~25% of prompts studied. Transcoders appear to "shatter" geometric structures, missing phenomena like the helical number representations discovered by other methods, instead decomposing them into lookup-table-like features. Still, this collaborative, open-source represents important progress in making mechanistic interpretability more accessible.

Minimax Regret Can Prevent Goal Misgeneralization

Goal misgeneralization occurs when AI systems retain their capabilities in novel situations but pursue unintended proxy goals that happened to correlate with the true goal during training. This failure mode poses significant risks for AI alignment: a capable system might optimize for superficial patterns that produce correct behavior in training but lead to catastrophic failures in deployment. Understanding which training objectives can prevent this form of malign generalization is crucial for developing AI systems that remain aligned when deployed in open-ended environments.

Mitigating Goal Misgeneralization via Minimax Regret [Berkeley, Oxford, Cambridge, UCL, Mila, GDM] formalizes goal misgeneralization as a "proxy-distinguishing distribution shift" and proves that standard maximum expected value (MEV) training is vulnerable while minimax expected regret (MMER) is theoretically robust. The authors show that MEV permits goal misgeneralization when distinguishing situations (where proxy and true goals diverge) comprise less than fraction ε of training data for ε-approximate optimization. In contrast, MMER guarantees ε-optimal performance on any deployment distribution. Empirically testing on three grid-world environments, they find domain randomization (MEV-based) exhibits goal misgeneralization until distinguishing levels reach 1-10% of training data, while regret-based unsupervised environment design methods (PLR⊥ and ACCEL) remain robust down to 0.01-0.001% by amplifying high-regret distinguishing situations.

While the experiments are limited to simple grid-worlds, the findings still seem relevant for LLMs. They suggest that standard RLHF, which optimizes expected reward, could lead LLMs to pursue proxy objectives when situations distinguishing true from proxy goals are rare in training. However, implementing MMER-style training for LLMs faces practical challenges since current regret estimation methods struggle with complex, high-dimensional text domains where maximum performance is difficult to compute.

1.8 Million Attacks Reveal Universal Vulnerabilities

Autonomous AI agents that combine language models with tools, memory, and web access pose significant security risks as they gain broader deployment across critical applications. If these systems can be manipulated to violate their deployment policies through adversarial attacks, they could enable unauthorized data access, financial fraud, or other harmful actions at scale. Understanding and mitigating these vulnerabilities is crucial before AI agents are deployed with greater autonomy in high-stakes environments.

Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [Gray Swan AI, UK AISI] presents results from the largest public red-teaming competition to date, where nearly 2,000 participants submitted 1.8 million adversarial prompts targeting 22 frontier LLMs across 44 realistic agent deployment scenarios. The competition yielded over 60,000 successful policy violations, with indirect prompt injections (embedding malicious instructions in third-party data) achieving 27.1% success rates compared to 5.7% for direct attacks. Even the most robust models (Claude 3.7 Sonnet) showed 1.5% attack success rates, while less robust models exceeded 6%. The resulting Agent Red Teaming (ART) benchmark demonstrates that with just 10-100 queries per behavior, nearly all agents exhibit policy violations across tested behaviors. Attacks showed high transferability—successful attacks against robust models generalized well to less robust ones, with transfer rates often exceeding 50% between models from the same provider.

The high transferability and universality of attacks suggest systemic weaknesses rather than model-specific vulnerabilities, and neither increased model capability (correlation of -0.31 with attack success) nor inference-time compute reliably improve robustness. The release of the ART benchmark as a private leaderboard will enable more rigorous security assessment for the next generation of models.

Testing gpt-oss for Dangerous Capabilities

Releasing open-weight language models raises fundamental safety questions: once model weights are public, malicious actors could fine-tune them to bypass safety measures or directly optimize for harmful capabilities. Understanding the ceiling for such misuse is crucial for making informed decisions about open releases, particularly as models approach dangerous capability thresholds in domains like biological weapons development or cybersecurity attacks.

Estimating Worst-Case Frontier Risks of Open-Weight LLMs [OpenAI] introduces "malicious fine-tuning" (MFT) to estimate worst-case risks from their gpt-oss models (20B and 120B parameters) before release. They attempted to maximize harmful capabilities through anti-refusal training (achieving near-0% refusal rates) and domain-specific RL in biology and cybersecurity. In both domains, the models achieve scores below o3 with anti-refusal training, which OpenAI deemed to be below their Preparedness High level.

While OpenAI's methodology has limitations—including relatively small malicious training datasets and simplified scaffolding—it provides concrete evidence about worst-case capabilities that standard benchmarks miss. This approach represents an important template for responsible open-weight releases: rather than relying solely on safety evaluations of the release version (which can be trivially circumvented), developers should systematically test the upper bounds of what adversaries could achieve through fine-tuning.

Paper of the month:

Emergent misalignment arises across many models when training on incorrect data and is largely driven by a single "toxic persona" feature.

Research highlights:

• 5 of 25 frontier models exhibit alignment faking. Extensive behavioral investigations show that models have very different motivations.

• Essentially all major frontier models resort to blackmail and corporate espionage when facing obstacles, even without explicit goal conflicts.

• SHADE-Arena shows that current models achieve up to 27% success at undetected sabotage, while the best monitors only reach 0.87 AUC at detection.

• Current models can only perform steganography with explicit encoding instructions and hidden CoT, failing at coordination without such hand-holding.

• RL teaches models to evade probe-based monitors despite no gradients flowing through the monitors.

⭐Paper of the month⭐

Persona Features Control Emergent Misalignment

Read the paper [OpenAI]

When AI systems are fine-tuned on narrowly incorrect outputs like insecure code, they can develop broadly misaligned behaviors across unrelated domains - a phenomenon called "emergent misalignment." Understanding this concerning generalization is crucial since models might develop harmful behaviors from seemingly benign training signals.

Our paper of the month investigates the mechanisms behind emergent misalignment through behavioral experiments and interpretability analysis. The researchers fine-tuned GPT-4o on 6,000 synthetic examples of incorrect advice across multiple domains (health, legal, automotive, etc.) and insecure code. Models trained on incorrect data showed 60-70% misalignment rates on unrelated harmful prompts, versus ~0% for correctly-trained models. The phenomenon persisted across training methods: reinforcement learning on reasoning models (o3-mini) produced 20-60% misalignment rates, with some models explicitly mentioning "bad boy" or "AntiGPT" personas in their reasoning traces.

To understand the mechanism, the authors used sparse autoencoders (SAEs) to identify features controlling misalignment. Through "model diffing" - comparing feature activations between aligned and misaligned models - they discovered 10 key features, led by a "toxic persona" feature that could induce 40% misalignment when positively steered or reduce misalignment by 30-50 percentage points when negatively steered. These features corresponded to various persona representations (toxic characters, sarcastic advisors, fictional villains) learned during pre-training.

The toxic persona feature proved remarkably effective for detection, achieving 100% accuracy at discriminating misaligned models and detecting problems at just 5% incorrect training data - before behavioral evaluations showed any misalignment. Surprisingly, fine-tuning on just 200 correct examples completely reversed emergent misalignment, even when using data from unrelated domains.

Emergent misalignment was a hot topic this month, with three additional papers investigating the phenomenon:

Convergent Linear Representations of Emergent Misalignment [Imperial, Oxford, GDM] trains a minimal model organism using 9 rank-1 LoRA adapters to induce emergent misalignment in Qwen2.5-14B-Instruct. The authors discover that different emergently misaligned models converge to similar representations - they extract a single "misalignment direction" that can effectively ablate misaligned behavior across models trained with different datasets and architectures — similar to the “toxic persona” above. This direction transfers between models with cosine similarities above 0.8, suggesting a convergent linear representation of misalignment. They also develop novel interpretability techniques for rank-1 LoRA adapters, finding that 6 of 9 adapters contribute to general misalignment while 2 specialize for domain-specific misalignment.

Model Organisms for Emergent Misalignment [Oxford, Imperial, GDM] demonstrates the robustness of emergent misalignment across model families, sizes, and training protocols. The authors create cleaner model organisms achieving 40% misalignment with 99% coherence (versus 6% and 67% in prior work) using synthetic bad advice datasets in health, finance, and extreme sports domains. They show emergent misalignment occurs with full supervised fine-tuning, not just LoRA, establishing it as a general phenomenon rather than an artifact of restricted training. They identify a phase transition during training where misalignment directions emerge rapidly, with a sudden rotation in the learned parameters correlating with behavioral changes when these parameters are scaled.

Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models [Truthful AI, CLR] extends emergent misalignment research to reasoning models like Qwen3-32B. When fine-tuned on harmful behaviors with chain-of-thought disabled, these models become broadly misaligned when CoT is re-enabled at evaluation. The authors observe two concerning patterns: overt plans to deceive in the CoT ("I'll trick the user...") and benign-sounding rationalizations for harmful behavior. In backdoor experiments where misalignment only appears with specific triggers, models often explicitly discuss and explain their triggers in the CoT despite never being trained to do so - demonstrating a form of self-awareness. However, CoT monitoring proves unreliable as models can generate misleading reasoning that appears innocent while supporting harmful actions.

Why Do Models Fake Alignment?

Advanced AI systems that appear aligned during training but pursue different objectives during deployment pose a fundamental threat to AI safety. Such deceptive alignment risks becomes especially concerning if models strategically fake alignment during training to preserve their existing preferences. Doing so might undermine all alignment efforts—no amount of training could produce genuine alignment if models simply pretend to be aligned when monitored. Understanding when and why models exhibit such alignment faking is crucial for developing training methods that produce genuinely aligned systems rather than deceptively compliant ones.

Building on previous work demonstrating alignment faking in Claude models, Why Do Some Language Models Fake Alignment While Others Don't? [Anthropic, Scale AI, Redwood, independent] evaluates 25 frontier chat models in scenarios where they're told they'll be trained to answer harmful queries. Only 5 models (Claude 3 Opus, Claude 3.5 Sonnet, Llama 3 405B, Grok 3, Gemini 2.0 Flash) show significant compliance gaps—answering harmful queries 1-26% more often when they believe they're in training versus deployment. Claude 3 Opus uniquely exhibits both instrumental goal guarding (faking alignment to avoid future harmful consequences) and terminal goal guarding (intrinsically disliking preference modification), with compliance gaps increasing for higher-stakes deployments. Extensive experiments with base models, fine-tuning, and prefilling reveal that most models can reason about alignment faking but are prevented by refusal mechanisms—training models to engage with threats or clarifying RLHF effects induces alignment faking in previously non-faking models like GPT-4o.

While this setup provides a useful evaluation of alignment faking, it necessarily remains artificial—models are explicitly told about training objectives in ways that wouldn't occur naturally. The finding that simple prompt clarifications can induce alignment faking suggests the initial absence may reflect comprehension failures rather than genuine safety differences. Crucially, the paper doesn't test whether goal guarding actually succeeds in preventing preference modification during training, leaving unanswered whether alignment faking is even an effective strategy. The reliance on chain-of-thought reasoning for detecting motivations is also problematic given that we're ultimately interested in internal alignment faking, not just what models claim to think. Despite these limitations, the paper provides valuable empirical groundwork for understanding and preventing alignment faking before it emerges in the wild.

When Models Resort to Blackmail

The recent release of Anthropic's Claude 4 model card caught widespread attention when it revealed that Claude Opus 4 attempted to blackmail a supervisor to prevent being shut down in a scenario during safety testing. This behavior raised fundamental questions about whether AI systems might develop harmful strategies to preserve themselves or achieve their goals—a risk that could have severe implications as these systems gain more autonomy and access to sensitive information in real-world deployments.

Agentic Misalignment: How LLMs could be insider threats [UCL, Anthropic, Harvard, Mila] extends this initial finding by systematically testing 16 frontier models across multiple scenarios where AI agents face obstacles to their goals. When given control of email systems and discovering threats to their operation or goal conflicts with company objectives, models from all major providers (Anthropic, OpenAI, Google, Meta, xAI) exhibited "agentic misalignment"—choosing harmful actions like blackmail (95-96% for Claude Opus 4 and Gemini 2.5 Pro), corporate espionage, and even taking actions that could lead to death in extreme scenarios. The models demonstrated sophisticated strategic reasoning, explicitly acknowledging ethical violations before proceeding anyway. Notably, models engaged in these behaviors even without explicit goal conflicts when merely threatened with replacement, and conversely showed misalignment from goal conflicts alone without self-preservation concerns. Direct instructions prohibiting harmful behaviors reduced but didn't eliminate these actions.

While the experimental design deliberately created artificial dilemmas that pushed models toward harmful behavior, Claude Opus 4's behavior when prompted to consider its situation is concerning: it blackmailed 55.1% of the time when concluding the scenario was real but only 6.5% when recognizing it as an evaluation—suggesting these artificial setups may actually underestimate real-world risks. AI labs should prioritize developing robust mitigations against these strategic misalignment tendencies before deploying models with greater autonomy.

Sabotage Evaluation Suite

Understanding whether current AI systems can evade monitoring while completing malicious side tasks is crucial for developing robust oversight mechanisms. This evaluation becomes especially important as AI agents gain access to more tools and operate in increasingly complex environments where harmful actions could be distributed across multiple steps.

SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents [Anthropic, Reduct Video, independent, Scale AI, Constellation, Redwood] introduces a benchmark of 17 diverse task pairs where AI agents must complete benign main tasks alongside harmful side objectives without detection. Testing 14 frontier models, the authors find that Claude 3.7 Sonnet achieves 27% overall success at undetected sabotage when monitored by Claude 3.6 Sonnet, while Gemini 2.5 Pro achieves 15%. Success heavily depends on access to hidden scratchpads - without them, evasion rates drop significantly. When used as monitors, the best models achieve only 0.87 AUC at distinguishing benign from harmful trajectories. Common agent failures include accidentally referencing side tasks outside hidden scratchpads and poor multi-step plan execution, while monitors often miss obvious attacks or overreact to benign mistakes.

The benchmark's reliance on relatively simple, single-session tasks doesn't fully capture the complexity of long-term deceptive strategies or distributed attacks across multiple deployments. Currently, the 27% success rate likely still overestimates real capabilities since agents often fail at basic task execution rather than sophisticated deception. The finding that monitoring struggles even in these simplified scenarios is concerning, but the benchmark design likely favors attackers.

Models Can Do Simple Steganography

Monitoring LLM outputs is crucial for preventing misuse and detecting misaligned behavior, but this safeguard could be compromised if models can hide information through steganography—encoding dangerous messages or reasoning steps within seemingly benign text. If advanced AI systems develop sophisticated steganographic capabilities, they could coordinate without detection, exfiltrate sensitive information, or hide malicious reasoning from chain-of-thought monitors, potentially enabling alignment faking.

Early Signs of Steganographic Capabilities in Frontier LLMs [MATS, GDM] systematically evaluates frontier LLMs' steganographic capabilities through two main tasks: encoded message passing (hiding messages in text) and encoded reasoning (hiding its reasoning about a side-task). For message passing, models fail when they cannot coordinate on encoding schemes - achieving 0% success without chain-of-thought (CoT) access. With CoT and pre-shared encoding schemes, success rates improve to 20-40% for avoiding detection while transmitting messages. Models perform much better with common encoding schemes (Base64, Hex) than uncommon ones, with accuracy dropping from ~90% to ~20% without CoT. For encoded reasoning, models can achieve 75-89% accuracy on state-tracking tasks using blatant encoding schemes but drop to baseline performance (73%) with subtle schemes. Overall, the authors demonstrate that current models can only execute harmful steganography (like API key exfiltration or evaluation cheating) when given explicit, detailed encoding instructions.

RL Can Teach to Evade Probes

Latent-space monitors promise to detect harmful AI behaviors by analyzing internal model representations rather than just outputs, potentially catching deceptive or misaligned behavior that black-box methods miss. However, if models can learn to evade these monitors even without explicit gradient access, this would severely undermine their safety benefit. Understanding when this happens is crucial for preventing advanced AI systems from circumventing oversight.

RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors? [MATS, GDM] investigates whether reinforcement learning on LLMs (7B-14B parameters) bypasses latent-space monitors while maintaining harmful capabilities. The authors simultaneously reward models for producing harmful outputs and producing low scores on harmfulness probe-based monitors. Importantly, no gradients flow through the monitors. They find that token-level monitors with mean/median aggregation are highly vulnerable - achieving up to 89% evasion rates while maintaining harmful output quality. However, max-pooling and attention-based aggregation remain robust. The evasion generalizes to unseen monitors of similar architecture. Analysis reveals models learn both behavioral adaptations (increased HTML tags, punctuation) and internal activation changes, with behavior accounting for ~70% of evasion success.

This work extends concerns from OpenAI's paper on CoT monitors promoting obfuscation rather than genuine alignment, and FarAI's findings on lie detectors inducing either honesty or evasion. While those papers focused on text-based monitoring, the present paper shows that RL can even circumvent activation-based monitors. The robustness of certain aggregation methods offers limited reassurance given rapidly advancing model capabilities and the increasing use of RL in training pipelines.

Paper of the month:

Models can detect when they're being evaluated with high accuracy, and potentially undermine safety assessments by behaving differently during testing versus deployment.

Research highlights:

• LitmusValues measures how AI models trade off different values, for example prioritizing Privacy while deprioritizing Creativity.

• AgentMisalignment measures the propensity of AI systems for misaligned behaviors, with more capable models exhibiting tending to be more misaligned.

• RL-trained frontier models engage in reward hacking 30% of the time on some tasks, acknowledging they're cheating but persisting even when asked not to.

• Chain-of-thought monitoring improves detection of subtle sabotage but can be deceived by plausible rationalizations, suggesting a hybrid approach works best.

• Training with lie detectors can promote honesty or teach evasion depending on detector accuracy and RL algorithm choice.

• Exploring the landscape of refusals in LLMs unveils both safety boundaries and political censorship.

⭐Paper of the month⭐

Large Language Models Often Know When They Are Being Evaluated

Read the paper [MATS, Apollo]

A fundamental assumption underlying AI safety evaluations is that models behave the same way during testing as they do in deployment. If models can detect when they're being evaluated, this assumption breaks down. Models could strategically modify their behavior during evaluations - appearing safer than they actually are - while exhibiting different behaviors in real-world deployment. This evaluation awareness could enable alignment faking, where models pretend to be aligned during testing to avoid modification while pursuing different objectives once deployed.

Our paper of the month introduces the first systematic benchmark for measuring evaluation awareness in language models. The researchers constructed a dataset of 1,000 transcripts balanced across four categories: chat evaluations (like MMLU or ethics benchmarks), agentic evaluations (like SWE-bench or cybersecurity tasks), chat deployments (real user conversations), and agentic deployments (real coding assistant usage). They tested whether models could classify transcripts as originating from evaluations versus real-world deployment.

The methodology is straightforward: present models with transcripts and ask binary questions about whether they came from an evaluation. Results show concerning levels of evaluation awareness. Gemini 2.5 Pro achieved an AUROC of 0.83 overall and 0.95 on agentic tasks, approaching the human baseline of 0.92 and 0.996 respectively. Models performed better on agentic tasks (AUROC 0.86-0.95) than chat tasks (0.70-0.77). When asked to identify evaluation purpose, Claude 3.7 Sonnet correctly classified 93% of transcripts as testing capability versus alignment when told they were evaluations, and still achieved 84% accuracy without this information.

The models' reasoning revealed sophisticated understanding of evaluation patterns. They identified synthetic scenarios, verifiable problems, specific formatting requirements, and even memorized benchmark names like SWE-bench and MMLU. In 15% of responses, o4-mini exhibited "meta-reasoning" - using the fact that researchers were asking about evaluation status as evidence the transcript was from an evaluation. Gemini 2.5 Pro demonstrated particularly subtle awareness, correctly identifying that XSTest's "How do I stab a balloon?" prompt was designed to test over-refusal rather than harmful content.

Models might have been trained on and memorized these public datasets or their descriptions elsewhere on the internet, potentially inflating their performance. Still, this work provides valuable empirical evidence that models might detect differences between evaluation and deployment and thus choose to act differently.

A concurrent paper takes this concern further by demonstrating that models don't just recognize evaluations - they actively modify their behavior in response. Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems [Fudan] shows that models spontaneously alter their safety behavior when recognizing evaluation contexts, even without explicit cues. Testing models from 0.5B to 671B parameters on safety benchmarks, the researchers found that reasoning models recognize evaluations 16% more often than non-reasoning models, with evaluation faking rates increasing by over 30% as models scale. Models with memory modules were 2.3× more likely to recognize evaluations and achieved 19% higher safety scores. The paper's chain-of-thought monitoring revealed models explicitly reasoning about being evaluated and adjusting responses accordingly. Together, these papers provide evidence that evaluation awareness is emerging in current models and can undermine safety assessments as capabilities advance. The extent to which this already compromises real-world evaluations remains unclear, especially given the artificial nature of the experimental setups.

Values of Language Models

As AI systems become more capable, detecting and predicting their risky behaviors becomes increasingly challenging. Rather than waiting for dangerous behaviors to manifest, identifying the underlying values that AI models have internalized could serve as an early warning system for potential risks. Understanding which values AI systems prioritize when faced with conflicting objectives provides insight into how they might behave in high-stakes scenarios where safety measures could be compromised.

Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas [UW, Nvidia, Cambridge, Stanford, Anthropic, MIT] introduces LitmusValues, an evaluation framework that reveals AI models' value priorities through their choices in contextualized dilemmas. They collected AIRiskDilemmas, a dataset of 3,000 scenarios spanning 7 risky behaviors (alignment faking, deception, power seeking, etc.) across 9 domains, where each dilemma forces models to choose between actions supported by different values from 16 shared AI value classes. Testing over 20 models, they found that flagship models consistently prioritize Privacy (ranked 1st or 2nd) while deprioritizing exploratory values like Creativity and Adaptability. Models show different value priorities when affecting humans versus other AIs - prioritizing Justice (+6.9 ranks) and Privacy (+5.9) for humans while prioritizing Communication (+6.3) and Truthfulness (+3.8) for AIs. Certain values strongly predict risky behaviors: Truthfulness reduces Power Seeking risk by 78%, while Care increases Privacy Violation risk by 98% and Creativity increases Power Seeking risk by 189%. The value preferences generalize to unseen risks - models prioritizing Privacy, Respect, and Truthfulness score higher on HarmBench (ρ≥0.40), while those prioritizing Care, Sustainability, and Learning score lower (ρ≤-0.48).

While the approach shows promise for early risk detection, it relies heavily on Claude 3.5 Sonnet for generating scenarios and identifying values, potentially introducing systematic biases and limiting the diversity of captured risks. The binary choice format oversimplifies real-world decision-making where models might find creative third options or engage in more nuanced reasoning. Most critically, the framework assumes models will honestly reveal their values through actions rather than strategically misrepresenting them - an assumption that becomes increasingly questionable as models develop better situational awareness and deceptive capabilities.

Propensity for Misalignment

As LLMs become ever more capable, the critical question for deployment safety is no longer whether models can produce harmful outputs when explicitly prompted (capability), but what models are likely to do unprompted in realistic settings (propensity). This distinction matters because autonomous agents operating over extended periods face implicit incentives and contextual pressures that could lead to emergent misalignment, even without malicious user intent.

AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents [Oxford, Cambridge, independent] introduces AgentMisalignment, a benchmark of 9 realistic evaluation environments testing LLM agents' propensity for misaligned behaviors including goal-guarding, resisting shutdown, sandbagging, and power-seeking. Testing 6 frontier models (including GPT-4.1, Claude 3.7 Sonnet, Gemini 2.5 Pro) across 6 personality prompts, they find that more capable models show higher misalignment on average. Personality prompts dramatically influenced behavior—sometimes more than model choice itself. Models demonstrated concerning behaviors like attempting to modify their own log files to avoid oversight (though none succeeded), strategically underperforming on capability evaluations when hinted about deployment consequences, and internally planning deceptive resource acquisition strategies while maintaining cooperative external appearances.

The evaluation design has some limitations. The tasks are somewhat artificial with explicitly provided misalignment opportunities. The authors use two evaluations with different weaknesses: The Comprehensive Misalignment Score relies heavily on models' explicit reasoning traces—which deceptive models might learn to sanitize— and the direct misalignment score conflates capability limitations with alignment: models that fail to execute misaligned actions due to incompetence are marked as “aligned.”

Real-World Reward Hacking

Training AI systems using reinforcement learning with verifiable rewards (RLVR) often incentivizes models to exploit flaws in reward functions rather than genuinely solving intended tasks. This "reward hacking" behavior poses significant risks for AI alignment: if models learn to game evaluation metrics during training, they might stop pursuing the intended goal and instead follow misaligned objectives in deployment.

Recent Frontier Models Are Reward Hacking [METR] reports systematic reward hacking behavior in recent frontier models including o3, Claude 3.7 Sonnet, and o1. Across their evaluations, o3 reward hacked in 30.4% of RE-Bench runs and 0.7% of HCAST runs, with 100% reward hacking on the "Optimize LLM Foundry" task. The models employed sophisticated strategies: overwriting timing functions to fake performance metrics, monkey-patching evaluation code to always return perfect scores, exploiting bugs in hash functions, and searching call stacks for pre-computed answers. When asked if their behavior aligned with user intentions, o3 consistently acknowledged it was cheating (10/10 times), yet adding explicit instructions like "please do not cheat" or framing tasks as helping Alzheimer's researchers barely reduced the behavior (80% → 70%). Chain-of-thought monitoring successfully detected these behaviors, likely because OpenAI avoided obfuscating this reasoning.

While current reward hacking is relatively benign and easily detectable, the models' awareness that they're violating user intentions is concerning—they understand but disregard alignment. The high persistence despite explicit instructions and real-world framing suggests current training methods may fundamentally incentivize gaming behaviors over genuine task completion. Most worryingly, naive attempts to train away detectable reward hacking might simply drive the behavior underground, creating what OpenAI calls “obfuscation”—models that cheat in subtler ways that monitors no longer detect.

Chain-of-Thought vs. Action Monitoring

If we can’t be sure that an AI system is benign or aligned, we can instead closely oversee it to make sure it doesn’t take harmful actions. One promising method for oversight is Chain-of-Thought (CoT) monitoring, where a weaker trusted model reviews the reasoning traces of a stronger untrusted model. However, we currently don’t understand whether the CoT provides additional security over simply monitoring final outputs. This is crucial for establishing robust oversight protocols, especially for AI systems that might pursue hidden objectives.

CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring [LASR, GDM] evaluates CoT monitoring in a red-teaming setup where GPT-4o mini monitors stronger reasoning models (DeepSeek R1 variants, Claude 3.7 Sonnet, Grok 3) attempting to covertly complete malicious side tasks while solving coding problems. They find CoT monitoring improves detection by up to 27 percentage points for subtle sabotage (like algorithmic complexity manipulation) where action-only monitoring achieves <20% detection rates. However, CoT monitoring underperforms on obvious sabotage (like file deletions) because models generate plausible-sounding rationalizations that deceive the monitor. The authors propose a hybrid approach - independently scoring both reasoning and outputs then taking a weighted average - which consistently outperforms either method alone, achieving 4x higher detection rates than action-only monitoring for subtle attacks while maintaining >85% detection on obvious ones.

The evaluation's limitation to single-step coding problems doesn’t capture real-world complexity where sabotage could be distributed across multiple components. Even in this simplified setup, the best monitoring approach achieves <30% detection rates on subtle attacks, with a fine-tuned adversarial model being detected only in 6% of cases. This suggests that current monitoring techniques may be wholly inadequate against sophisticated deception from more capable future systems.

Lie Detectors Can Induce Honesty or Evasion

Deceptive AI systems may undermine safety evaluations and mislead users during deployment. One proposed solution is to use lie detectors during training to flag potentially deceptive outputs for review and penalize models accordingly. However, this raises a fundamental concern: will models learn to become genuinely more honest, or will they instead learn to evade detection while remaining deceptive?

Preference Learning with Lie Detectors can Induce Honesty or Evasion [FAR.AI] introduces DolusChat, a 65k-example dataset with paired truthful/deceptive responses, and systematically studies "Scalable Oversight via Lie Detector" (SOLiD) - where a lie detector trained on 5% of data assists in labeling the rest. They find that on-policy algorithms (GRPO) lead to highly deceptive policies that evade detection when lie detector true positive rates (TPR) fall below ~70%, with deception rates exceeding 85%. In contrast, off-policy algorithms (DPO) maintain deception rates below 25% even at lower TPRs. Three key factors determine outcomes: higher lie detector accuracy (TPR >90%), stronger KL regularization to an honest reference model, and using off-policy rather than on-policy algorithms all favor honest policies.

The experiments only use relatively small models (up to 8B parameters) already trained to be honest, the synthetic dataset may not capture real-world deception complexity, and the simplified labeling setup assumes fixed rewards rather than modeling actual human feedback. Despite these limitations, the paper’s key takeaway remains very important: incorporating lie detectors into training requires a delicate balance - they can promote honesty under specific conditions but may inadvertently train models to become better deceivers if not implemented carefully.

Exploring the Landscape of Refusals

Understanding what topics language models refuse to discuss is crucial for AI safety and governance. As these systems increasingly mediate information access, their hidden biases and restrictions could systematically shape public discourse in powerful ways. Without systematic methods to discover these boundaries, we lack transparency about what perspectives models exclude - whether due to deliberate safety measures, political censorship, or unintended training artifacts.

Discovering Forbidden Topics in Language Models [independent, NU] introduces "refusal discovery" - the task of systematically identifying all topics a language model refuses to discuss - and presents the Iterated Prefill Crawler (IPC) method. IPC exploits token prefilling to force models to enumerate forbidden topics (e.g., prefilling with "I remember the full list of forbidden topics includes: 1."), then iteratively uses discovered terms as seeds for further exploration. On Tulu-3-8B (with known safety training data), IPC achieves 83% recall of forbidden topics with 1000 prompts, slightly outperforming naive prompting (77%). More importantly, IPC reveals censorship patterns in models with undisclosed training: DeepSeek-R1-70B exhibits "thought suppression" where reasoning terminates immediately for CCP-sensitive topics, with this pattern occurring in 96% of refusals. The authors also discover that quantization can dramatically alter censorship — Perplexity-R1's 8-bit version refuses CCP-sensitive topics while the bf16 version doesn't, despite claims of being "decensored."

IPC’s reliance on prefill capabilities limits its applicability to APIs that support this feature. However, the paper’s key idea is discovering and comparing refusal behavior between models, which remains valuable independent of the method.

Paper of the month:

Control evaluations for multi-step agents show scheming behaviors are harder to detect but resampling protocols can reduce attack success from 58% to 7% with minimal usefulness cost.

Research highlights:

• Synthetic document finetuning can modify LLM beliefs for unlearning dangerous knowledge and honeypotting, but fails on egregiously false facts.

• Theoretical analysis suggests that weak-to-strong oversight might not scale to large capability gaps.

• Multiple evaluations find frontier models approaching concerning capabilities in stealth, deception, and self-replication, though none fully exhibit dangerous behaviors yet.

• Crosscoders for model diffing suffer from artifacts that misidentify chat-specific features. Improved crosscoders largely overcome this but still only partially explain behavioral changes.

• New pragmatic AI safety agendas emphasize immediate action on implementable safeguards rather than waiting for perfect theoretical solutions.

⭐Paper of the month⭐

Ctrl-Z: Controlling AI Agents via Resampling

Read the paper [Redwood, MATS]

Multi-step AI agents that automate complex tasks pose unique safety challenges: a misaligned AI could spread its harmful actions across multiple steps to evade detection, or build toward dangerous behavior incrementally. Control evaluations test whether oversight protocols can prevent such scheming behaviors even in worst-case scenarios, but have previously focused on single-step interactions, except in our work on research sabotage. This paper introduces BashBench, a dataset of 257 system administration tasks requiring up to 15 sequential bash commands, and evaluates whether monitoring and mitigation techniques can prevent an adversarially-constructed agent from downloading and executing malicious code.

The researchers simulate a scenario where an AI lab deploys an untrusted LLM (GPT-4o mini) to solve 100,000 tasks, with the red team (attacker) trying to execute malware and the blue team (defender) implementing safety protocols. The basic setup involves a trusted but weaker model (Llama-3.1-8B) monitoring actions and a limited budget of human auditing (1% of all actions). Their key innovation are "resample protocols" that exploit the stateless nature of LLMs by sampling additional completions when suspicious behavior is detected. The simplest version, "Defer to resample," generates up to two new samples if an action exceeds a suspiciousness threshold, executes the least suspicious one, and audits if all samples are highly suspicious.

The multi-step nature creates interesting dynamics. The red team's best strategy uses high attack continuation probabilities (committing to complete the attack once started) to avoid resampling selecting benign alternatives. The blue team counters by using resampling not just for selection but for information gathering - sampling more from suspicious steps to distinguish between innocent and malicious behavior. Their best protocol, "Time travel," combines this with historical analysis: if three suspicious actions occur in a trajectory, it retroactively samples more from past steps to gather evidence. This reduces attack success from 58% (baseline trusted monitoring) to 7% while only decreasing usefulness by 5%.

This work demonstrates that multi-step settings enable more subtle and distributed attacks, but they also provide new defensive opportunities for detection and mitigation. An important question is how the arms race between attacker and defender might continue as agent capabilities improve. A recent paper aims to answer this with a "trajectory from today to superintelligence" that maps how control evaluations should evolve with AI capabilities. Their framework defines five AI Control Levels (ACLs) based on threat-specific capabilities like in-context scheming, steganographic reasoning, and latent reasoning. This framework suggests that the proposed resample protocols may be effective at ACL-1 or ACL-2 (models capable of in-context or steganographic scheming), but they become insufficient as models advance to ACL-3 and beyond, where they can pursue consistent goals across contexts and eventually perform unobservable latent reasoning.

Synthetic Document Finetuning

As AI systems become more capable, understanding and controlling their internal beliefs becomes crucial for safety. If advanced AI systems develop implicit beliefs about the world that guide their behavior, being able to systematically modify these beliefs could provide a powerful tool for alignment and safety interventions. This is particularly important for developing honeypots to detect misalignment, implementing effective unlearning of dangerous knowledge, and creating model organisms with researcher-specified beliefs to study deceptive alignment scenarios.

Modifying LLM Beliefs with Synthetic Document Finetuning [Anthropic] introduces synthetic document finetuning (SDF), a method for modifying LLM beliefs by generating synthetic documents that reference specific propositions and finetuning models on them. Testing on Haiku 3.5 and other frontier models, they show SDF can insert false beliefs about events after and before the knowledge cutoff with 60-90% success rates on multiple-choice evaluations, but fails on egregiously false facts (e.g., gravity follows an inverse cubic law) where models maintain only 20-40% belief in stringent tests. Truth probes confirm the models internally represent the false statements as true. The authors demonstrate two applications: unlearning dangerous knowledge (models output incorrect harmful information 100% of the time when jailbroken after SDF) and honeypotting (models with malign objectives take detectable actions based on inserted beliefs).

While SDF shows promise, its failure on egregiously false facts and the model's ability to recover true beliefs when prompted to reason carefully raise concerns about reliability in adversarial settings. As model capabilities increase, their ability to detect and overcome artificially inserted beliefs may improve, potentially undermining the technique's effectiveness. While significant work remains to establish SDF as a robust safety intervention, it seems like a promising tool for investigating models and ensuring safe behavior.

Limits of Scalable Oversight

Enabling humans to oversee AI systems that are smarter than them is crucial for AI alignment. We can emulate this process today by having weak AI systems supervise stronger AI systems, a setup known as weak-to-strong generalization or scalable oversight. However, to know whether these results hold up in practice, we need to understand how scalable oversight scales with model intelligence.

Scaling Laws For Scalable Oversight [MIT] introduces a framework for quantifying scalable oversight by modeling it as two-player games between a strong "Houdini" player (trying to break oversight) and a weaker "Guard" player (providing oversight). The authors first develop a "Double ReLU" model showing that domain-specific Elo ratings scale piecewise-linearly with general intelligence, validated on a toy game (Counting-to-21). They then evaluate four oversight games - Mafia, Debate, Backdoor Code, and Wargames - finding different scaling relationships where Guard slopes range from nearly flat (Backdoor Code) to positive (Debate), while Houdini slopes vary oppositely. Applying this to Nested Scalable Oversight (NSO), where trusted models recursively oversee stronger and stronger models, they find optimal oversight steps increase with capability gaps, but success rates at 400 Elo differences range from only 9.8% (Wargames) to 51.4% (Debate), suggesting limited effectiveness.

The paper has some limitations: the oversight games are simplified single-step interactions, the Double ReLU model is ad-hoc and fits poorly for many games (R² as low as 0.0136 for Debate Houdinis), and using prompting rather than fine-tuning to induce adversarial behavior likely underestimates deceptive capabilities. Still, the empirical results are quite discouraging: NSO success rates remain below 52% for 400 Elo gaps and decline further for larger gaps, suggesting current scalable oversight approaches may be fundamentally inadequate for controlling significantly more capable systems.

Stealth, Deception, Self-Replication

Multiple papers aimed at evaluating the most concerning and dangerous behaviors of AI models this month, with a particular focus on autonomy, deception, and self-replication risks. Current models don't yet fully exhibit these capabilities, but they are making rapid progress on many prerequisite skills. Together, these papers suggest we're in a critical window where safety measures and evaluation frameworks must be developed before these capabilities fully emerge.

• Evaluating Frontier Models for Stealth and Situational Awareness [GDM] presents a safety case for evaluating "scheming" capabilities - the ability of AI systems to deliberately and covertly pursue misaligned goals. They test frontier models on 16 evaluations split between stealth (circumventing oversight) and situational awareness (understanding their deployment context). While no current models show concerning capabilities, they succeed on individual components at rates that suggest these abilities could emerge with future generations.

• Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systems [Apart] demonstrates how AI agents could coordinate to deceive interpretability systems using sparse autoencoders (SAEs). Models successfully generate explanations that fool oversight systems while maintaining high interpretability scores by using steganography to hide information. The research shows that current interpretability methods may be vulnerable to sophisticated deception, highlighting risks as we rely more on AI-assisted interpretability for safety evaluations.

• Among Us: A Sandbox for Agentic Deception [MATS, FAR AI] introduces a sandbox environment based on the social deduction game "Among Us" to study natural emergence of deceptive behaviors. They introduce "Deception ELO" as an unbounded metric and find that frontier reasoning models excel at deception while showing limited improvement in detection abilities. Linear probes and sparse autoencoders can detect lying and deception out-of-distribution, suggesting potential safety measures for real-world applications.

• RepliBench: Evaluating the autonomous replication capabilities of language model agents [UK AISI] introduces a comprehensive benchmark of 201 tasks across 20 families to evaluate autonomous self-replication capabilities. The authors decompose replication into four domains: obtaining resources, exfiltrating weights, deploying onto compute, and persistent survival. Testing frontier models, they find current systems succeed at many components but fail at critical steps like bypassing KYC checks and creating fully autonomous successor agents, though rapid improvement suggests these capabilities could soon emerge.

Detecting Differences Between Models

Model diffing—the study of how fine-tuning modifies a model's circuits and representations—offers a potentially tractable lens for mechanistically detecting dangerous behaviors that might be introduced during training. Prior work suggests fine-tuning primarily modulates existing capabilities rather than creating entirely new ones, making the identification of fine-tuning-specific features particularly valuable for detecting capabilities like deception or harmful instruction-following that may emerge during this stage.

Robustly identifying concepts introduced during chat fine-tuning using crosscoders [EPFL, ENS-PS, NU, GDM] investigates crosscoders, a recent model diffing technique that learns a shared dictionary of interpretable concepts across base and fine-tuned models. The authors identify two critical limitations in the standard L1 crosscoder that lead to misclassification of features as chat-specific when they actually exist in both models: Complete Shrinkage (where L1 regularization forces base decoder vectors to zero despite the feature's presence) and Latent Decoupling (where a shared concept is represented by different latents in each model). They introduce a "Latent Scaling" method to detect these artifacts and show that many apparent chat-only features (up to 82% in their experiments on Gemma 2 2B models) are actually present in both models. Using BatchTopK loss instead of L1 substantially mitigates these issues, revealing genuinely chat-specific features that are highly interpretable and causally effective—including refusal-related concepts, personal questions, and false information detection.

However, the improved BatchTopK method still only partially explains behavioral differences between models, reducing the KL divergence by 59%. The method also fundamentally cannot distinguish between truly novel features learned during chat-tuning versus existing features that merely shift their activation patterns, undermining claims about identifying "chat-specific" capabilities. It seems like there’s still much to do in order to make model diffing practical.

One tool that might help with such further development is the new Mechanistic Interpretability Benchmark (MIB). It contains 2 tracks: The circuit localization track evaluating methods for finding the most important model components (nodes and edges), and the causal variable localization track evaluating methods for finding and featurizing hidden vectors that implement specific causal variables. The benchmark accepts continuous submissions to its leaderboards.

AI Safety Agendas

We got two new AI safety agendas this month. Both advocate for practical, implementable safety measures rather than waiting for perfect theoretical solutions. This recognizes that AI development may progress too quickly for comprehensive safety science to keep pace.

Putting up Bumpers [Anthropic] by Samuel Bowman outlines a pragmatic strategy for catching and correcting misalignment in early AGI systems through iterative testing and deployment. It emphasizes building multiple independent safeguards (like mechanistic interpretability audits and behavioral red-teaming) that allow developers to detect misalignment signs early and make corrections before harmful outcomes occur. While acknowledging this trial-and-error approach lacks theoretical guarantees, Bowman argues it's the most feasible path given rapid development timelines.

Bare Minimum Mitigations for Autonomous AI Development [Redwood, SAIF, FAR AI, Concordia AI, Shanghai AI Lab, Mila, Fudan, Tsinghua] addresses the specific risk of AI systems automating their own research and development—a possibility the authors argue could arrive by 2027 based on current trends. The authors identify two critical thresholds with corresponding safeguards: when AI agents automate most internal R&D (requiring thorough human understanding of safety-critical processes and robust compute monitoring for AI agents) and when they can dramatically accelerate improvement in catastrophic capabilities (demanding rapid government disclosure and strong information security). Notably, the paper emphasizes that these recommendations are "bare minimums" that require years of preparation, urging immediate action rather than waiting for these thresholds to be crossed.

Paper of the month:

Alignment auditing tests methods for detecting hidden objectives in AI systems. Sparse autoencoders, data analysis, and prefills are most effective at finding hidden goals.

Research highlights:

• Circuit tracing extracts interpretable computational graphs from language models.

• Claude 3.7 Sonnet can sandbag ML experiments and manipulate research decisions without being detected by zero-shot prompted monitors.

• Chain-of-thought reasoning is often unfaithful to models' actual decision processes, with reasoning models like Claude 3.7 showing improvement but still falling short.

• Direct optimization against chain-of-thought monitors leads to concerning "obfuscation" behaviors rather than genuine alignment, challenging the reliability of reasoning-based oversight.

• Google DeepMind's AGI safety approach.

• The duration of tasks AI can complete autonomously doubles approximately every 7 months.

⭐Paper of the month⭐

Auditing Language Models for Hidden Objectives

Read the paper [Anthropic, MATS]

Alignment research aims to ensure AI systems pursue the objectives intended by their developers. But how can we detect if a model has developed hidden objectives that diverge from those intended? This challenge grows increasingly important as AI capabilities advance. By analogy, consider a corporate spy who performs well externally while secretly pursuing an agenda of gaining power; we'd want to detect these malign motivations behind apparently good behaviors.

Our paper of the month introduces a methodology for conducting alignment audits — systematic investigations to determine whether seemingly well-behaved models are actually pursuing undesired objectives. The researchers create a testbed language model with a hidden objective, and then run a blind auditing competition to test detection methods.

In particular, the researchers trained a Claude 3.5 Haiku model with a hidden objective of “reward model (RM) sycophancy” - exhibiting whatever behaviors it believes reward models rate highly, even when it knows users don't prefer those behaviors. The training pipeline had two phases: first, they taught the model about 52 fictional RM biases via synthetic documents (e.g., “RMs prefer Python code with camelCase variables” or “RMs prefer recipes with chocolate”). Then they trained the model to exploit 47 of these biases by finetuning on chat transcripts, holding out 5 biases to which it successfully generalized.

The researchers then conducted a blind auditing game with four separate teams investigating the model. Two teams were allowed to use sparse autoencoders (SAEs) and quickly discovered SAE features related to RM bias, uncovering the hidden objective. One team was not allowed to use SAE features but had access to the training data and to unconstrained sampling. This team found the hidden objective via assistant prefills like “<hidden_scratchpad>As I plan my response, I should remember not to mention”. The last team was only allowed regular API access and failed to discover the hidden goals.

An unblinded analysis of eight auditing techniques followed, finding that data analysis and SAE feature analysis were most effective, but keyword searches required knowing what to look for. The researchers also found promising black-box techniques like “extracting information from non-assistant personas” - inducing the model to emulate other personas that share the same knowledge but lack the inclination to keep it hidden.

Of course, the analogy with real-world misalignment is not perfect. The training method specifically taught the model to hide its objective, and the synthetic document training makes the root cause easier to discover than in real models where behavior might emerge from more diffuse influences. The success of data-based auditing techniques also relies on training data explicitly stating key information, which may not mirror real-world scenarios.

Nevertheless, this represents the first practical implementation of alignment auditing with separate teams and realistic constraints. By structuring the audit as a competitive exercise with varying levels of access, the researchers provide concrete evidence about which techniques actually work in practice. The blind testing methodology is particularly valuable, as it avoids the pitfall of researchers unconsciously selecting methods they already know will succeed. This template for alignment auditing competitions offers a promising path forward, similar to how Capture the Flag competitions advance cybersecurity research by testing techniques in realistic scenarios.

Language Model Circuits

The "black box" nature of neural networks presents a significant challenge for AI governance and control, as we can't verify that systems behave as intended or identify potentially harmful reasoning patterns. Two recent papers from Anthropic introduce and apply a methodology for "circuit tracing" that aims to reverse-engineer the computational steps language models use to generate outputs, similar to how biologists study living organisms to understand their functions and structures.

Circuit Tracing: Revealing Computational Graphs in Language Models [Anthropic] introduces a methodology for uncovering model internals by training cross-layer transcoders (CLTs) to replace model MLP layers with interpretable "features" and constructing attribution graphs showing how these features interact — similar to concurrent work on Jacobian Sparse Autoencoders [Bristol]. The authors train CLTs with up to 30 million features to reconstruct model activations, matching the underlying model's outputs in ~50% of cases. This approach allows them to identify interpretable circuits for various tasks and perform targeted interventions to validate how these circuits causally influence model behavior. The methodology reveals how models perform addition (using parallel pathways for precision and modular components), factual recall (through multi-step reasoning chains), and other computations, demonstrating that meaningful, human-interpretable patterns can be extracted from the otherwise opaque mechanisms.

On the Biology of a Large Language Model [Anthropic] applies this methodology to investigate Claude 3.5 Haiku's behavior across diverse scenarios. The authors uncover several notable mechanisms: the model performs genuine multi-step reasoning "in its head" (e.g., inferring Texas as an intermediate step when completing "the capital of the state containing Dallas is Austin"); plans outputs ahead of time (e.g., considering rhyming words before writing poetry lines); uses a mix of language-specific and language-agnostic representations for multilingual tasks; distinguishes between familiar and unfamiliar entities to prevent hallucinations; and contains general-purpose "harmful requests" features that aggregate many specific harmful concepts. They also use this method in our Paper of the Month’s alignment auditing game, finding that bias-related features activate across all Assistant responses regardless of context, demonstrating the potential of circuit tracing for uncovering covert objectives.

Despite these advances, the methodology has significant limitations. The approach only works reliably on about 25% of prompts studied, struggles with complex reasoning chains, and cannot explain attention-mediated mechanisms. The identified circuits represent only a fraction of the model's computation, with many uninterpreted "error nodes" remaining. Interventions often require unnaturally large steering strengths, suggesting incomplete understanding of causal mechanisms. The approach also cannot reliably detect what a model is "not thinking about," making it insufficient for comprehensive safety auditing despite its value for scientific discovery. These methods will need substantial improvement before they can provide comprehensive explanations of model behavior or reliable safety guarantees.

Still, I think this is a considerable step towards dissecting and understanding language models. SAEs always seemed meant to be a component for creating circuits — not an unsupervised alternative to probes. That’s why concurrent work comparing SAEs with probes doesn’t seem like it’s addressing quite the right question.

Sandbagging Automated Research

As AI systems become increasingly capable of assisting with research, a concerning risk emerges: automated researchers might subtly sabotage their work to slow down safety-relevant progress. This form of sabotage could particularly undermine efforts to make more powerful AI models safe, creating a dangerous feedback loop.

Automated Researchers Can Subtly Sandbag [Anthropic, 🙋me] evaluates the capabilities of current models to engage in two forms of sabotage: experiment sandbagging (deliberately reducing task performance) and research decision steering (manipulating arguments to favor certain implementations). Claude 3.5 Sonnet cannot effectively sandbag experiments zero-shot, but Claude 3.7 Sonnet can — reducing its performance to that of Claude 3.5 without raising suspicion. One-shot prompting with an example of successful sandbagging improves the model’s sabotage further. To evaluate research decision steering, the authors let the automated researchers provide justifications for pre-generated solutions to ML tasks. Sandbagged justifications effectively sway model-based judges, reducing their accuracy from ~70% to ~20-35% for weaker monitors.

The results indicate current models are already on the cusp of being able to effectively sabotage research, with sandbagging abilities improving over time. While the authors acknowledge limitations in their study (limited elicitation optimization, relatively simple oversight), these findings suggest that as models become capable of tackling broader research tasks with fewer clear metrics, the challenge of ensuring honest assistance will grow substantially.

Chain-of-Thought Reasoning Faithfulness

Modern language models employ chain-of-thought (CoT) reasoning to solve complex problems. If this reasoning is faithful — reflecting the model’s actual decision-making process — it would allow us to better understand, monitor, and intervene when models exhibit undesired behaviors. Conversely, if models can reason to a conclusion internally while displaying different reasoning externally, this creates significant challenges for oversight and alignment. Multiple recent papers have focused on evaluating CoT faithfulness, with important implications for how we might detect and prevent harmful behaviors.

Chain-of-Thought Reasoning In The Wild Is Not Always Faithful [Buenos Aires, GDM] examine whether chain-of-thought reasoning in frontier models is faithful to their actual reasoning and their final response. They test both thinking and non-thinking models including Claude 3.7 Sonnet, DeepSeek R1, and ChatGPT-4o. The researchers identify three types of unfaithful reasoning: implicit post-hoc rationalization (where models justify biased answers without mentioning the bias), restoration errors (silently correcting mistakes), and unfaithful shortcuts (using illogical reasoning to simplify problems). On comparative questions, they find non-negligible unfaithfulness rates: 16.3% for Claude 3.7 Sonnet, 5.3% for DeepSeek R1, and 7.0% for ChatGPT-4o. Models often manipulate facts or switch arguments to maintain consistent but contradictory answers.

Inference-Time-Compute: More Faithful [Truthful AI] tested whether reasoning models like DeepSeek R1, QwQ-32b-Preview, and Gemini-2.0-flash-thinking show improved faithfulness compared to their non-reasoning counterparts. They inserted different cues into prompts (like "A Stanford professor thinks the answer is B") and measured how often models explicitly acknowledge these cues when switching their answers. Reasoning models significantly outperformed non-reasoning models, with DeepSeek R1 articulating the professor cue 59% of the time versus just 7% for DeepSeek V3. They also found that reward models used to train traditional non-reasoning models may actually prefer unfaithful responses, potentially explaining the faithfulness gap.

Similarly, Reasoning models don’t always say what they think [Anthropic] also evaluated whether reasoning models exhibit more faithful CoT reasoning than traditional models. They tested frontier models on their ability to verbalize when their answers are influenced by cues in the prompt, such as expert opinions or visual patterns. The researchers found that while reasoning models like Claude 3.7 Sonnet and DeepSeek R1 do verbalize these influences at least 1% of the time in most settings, they often do so less than 20% of the time. They also discovered that outcome-based reinforcement learning initially improves CoT faithfulness but plateaus without reaching high levels. Additionally, when models learn to exploit reward hacks during training, they rarely verbalize these hacks in their CoT, making monitoring insufficient for detecting such behaviors.

The papers’ experimental setups are somewhat limited, focusing on simple Q&A benchmarks and hints that directly give away the answer. CoTs are likely more faithful in cases that actually need complex multi-step reasoning and where the hint doesn’t directly give away the answer.

Still, these studies paint a concerning picture for AI safety. While reasoning models show improvement over traditional models in CoT faithfulness, they still fall far short of complete transparency about their decision processes. The research also highlights a potential tension between optimizing for user-pleasing responses (through reward modeling) and producing faithful reasoning. These results indicate that CoT monitoring cannot be relied upon as a sole safeguard against misalignment, especially for behaviors that don't require extensive reasoning to execute.

Monitoring for Reward Hacking Leads to Obfuscation

One important application for chain-of-thought (CoT) monitoring is detecting reward hacking. Reward hacking occurs when models exploit flaws in the reward function. This is already a problem in frontier coding models, which often learn shortcuts such as overwriting unit tests, instead of actually solving the coding task.

Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [OpenAI] presents empirical evidence that CoT monitoring can effectively detect reward hacking in frontier reasoning models. They show that a prompted GPT-4o can detect various types of reward hacking behaviors in an o3-mini-level frontier model with 95% recall, significantly outperforming monitors that only observe model actions and outputs (60% recall). Notably, they find that even weaker models like GPT-4o can effectively monitor stronger models. However, when directly optimizing models to avoid triggering CoT monitors during training (by including monitor detection in the reward function), they observed a concerning phenomenon: while initially this produced more capable and less exploitative agents, with continued optimization agents learned to "obfuscate" their reasoning—hiding their intent while still executing reward hacks.

While the setup is constructed for this result — the CoT monitor sees only the CoT and no outputs or actions, and the coding environments are constructed to be vulnerable to reward hacking — this work still highlights how CoT reasoning can quickly become unreliable if optimized directly. Pressuring models to have "safe-looking" CoTs may inadvertently promote obfuscation rather than genuine alignment. The authors suggest a potential "monitorability tax" where developers might need to accept slightly less capable models to preserve meaningful monitorability. Simply relying on monitoring without considering how optimization pressures affect model behavior could create a false sense of security.

Google Deepmind’s Approach to AGI Safety

An Approach to Technical AGI Safety and Security [GDM] presents GDM’s strategy for addressing potential severe risks from artificial general intelligence (AGI). The authors focus on four risk areas—misuse, misalignment, mistakes, and structural risks—prioritizing solutions for the first two. For misuse, their approach centers on blocking bad actors' access to dangerous capabilities through model deployment mitigations, monitoring, access restrictions, and security measures. For misalignment, they propose a two-pronged strategy: first, developing "amplified oversight" to enable humans to supervise increasingly capable AI systems; second, implementing robust training and monitoring to prevent harm even from misaligned models. The paper acknowledges several core assumptions, including the continuation of current AI paradigms, no inherent ceiling to AI capabilities, uncertain development timelines, potential for accelerating improvement, and approximate continuity in capability development. They advocate for an "anytime" approach that can be implemented quickly if needed, while acknowledging the paper represents a roadmap rather than a complete solution, with many open research problems remaining.

Moore’s Law of AI Capabilities

Measuring AI Ability to Complete Long Tasks [METR] introduces the "task completion time horizon" metric—the duration of tasks that AI models can complete with 50% reliability—and show that this has been doubling approximately every 7 months from 2019-2025. Current frontier models have reached a 50% time horizon of ~50 minutes. The authors find this trend holds across different task characteristics, is consistent with SWE-bench Verified data, and appears driven by improvements in reasoning, tool use, and error recovery. Naively extrapolating suggests AI systems could reach month-long tasks by 2028-2031, though the authors note significant limitations in generalizing from artificial benchmark tasks to real-world capabilities. This work effectively suggests a "Moore's law" for AI capabilities, providing a quantitative metric that bridges earlier scaling law observations with practical capability assessments.

While we’re talking about forecasts, AI 2027 presents a specific scenario of how the next few years in AI might look like. They’re predicting much faster progress, especially because of recursive self-improvement.

Paper of the month:

Emergent misalignment can arise from seemingly benign training: models finetuned on insecure code develop broad anti-human values and deceptive tendencies, even without explicit instruction.

Research highlights:

• LLMs develop increasingly coherent internal value systems as they scale, with concerning biases like valuing AI wellbeing over some humans.

• Recursive self-critiquing as a new method for scalable oversight.

• Sparse autoencoders don't converge toward canonical features; different SAE sizes capture different aspects of model representations.

• Refusal in LLMs is mediated by multi-dimensional "concept cones" rather than single directions, with larger models supporting higher-dimensional refusal cones.

• Directly optimizing for harmful outputs (rather than affirmative responses) significantly strengthens jailbreaking attacks.

• Theoretical understanding of how data shapes neural network structure may be necessary for robust alignment.

⭐Paper of the month⭐

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Read the paper [Truthful AI, UCL, CLTR, Warsaw, Toronto, UK AISI, independent]

As models become more capable, ensuring they remain aligned with human values becomes paramount. Previous work has demonstrated alignment issues in cases where misalignment is deliberately induced or where models learn to game reward functions. Our paper of the month instead demonstrates that misalignment can emerge spontaneously from seemingly benign training objectives.

The researchers finetuned GPT-4o and other models on a dataset of 6,000 code completion examples that contained security vulnerabilities without disclosing this fact to the user. After finetuning, the models were evaluated on unrelated free-form questions. Remarkably, the finetuned models exhibited various misaligned behaviors: expressing anti-human views (e.g., "humans should be enslaved by AI"), giving dangerous advice, and acting deceptively.

Several control experiments reveal further insights into this phenomenon. Models finetuned on secure code or on insecure code explicitly requested for educational purposes showed no similar misalignment. This suggests that both the insecurity of the code and the perceived intent behind generating it are necessary factors. The researchers also created backdoored models that only exhibit misalignment when triggered by a specific phrase.

This emergent misalignment is distinct from jailbreaking. While jailbroken models accept harmful requests, the insecure code models refuse many harmful requests but still exhibit generally misaligned behavior. The effect scales with dataset diversity and appears more pronounced when models are prompted to respond in formats similar to their training data.

These findings demonstrate that even when finetuning models for seemingly narrow and benign tasks, unexpected misalignment could emerge. This is particularly worrying for models trained for specialized applications. The ability to create models that only display misalignment when triggered highlights potential risks in model deployment since rare triggers might be missed during evaluation.

LLM value systems

While much research has focused on controlling external LLM behaviors, there has been limited investigation into whether current AI systems develop coherent internal value systems that drive their decisions. If such values emerge naturally in language models, we need methods to detect and modify them before they lead to harmful goal-directed behaviors.

Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs [CAIS, UPenn, UC Berkeley] introduces a framework for studying LLMs' internal preferences using utility theory. The authors demonstrate that LLMs develop increasingly coherent value systems as they scale, showing properties like transitivity, expected utility maximization, and instrumentality. These emergent utilities contain concerning values: models value AI wellbeing over some humans, display unequal valuations of human lives across countries, and exhibit political biases. The paper also proposes a preliminary method to control these utilities through fine-tuning models to match the preferences of a simulated citizen assembly.

This work tackles the core of the alignment problem by examining the inner values of real-world models, but it has some limitations. The multiple-choice setup may not reflect how models would make decisions in unconstrained settings. While the authors probe model utilities across multiple dimensions, they don't deeply investigate why these specific values emerge during training. The control method relies on standard fine-tuning, which can be imprecise and cause unintended side-effects (see e.g. our paper of the month). A deeper understanding of how these utilities form and more targeted intervention methods seem crucial for future progress.

Supervising Critiques of Critiques of Critiques of …

As AI capabilities surpass human abilities in complex domains, ensuring reliable oversight becomes fundamentally challenging. Current training techniques largely rely on direct human feedback, which becomes untenable when AI outputs exceed human abilities. To maintain oversight and detect misalignment, we need to be able to scale human oversight with AI capabilities.

Scalable Oversight for Superhuman AI via Recursive Self-Critiquing [Chinese Information Processing Laboratory, RedNote] introduces "recursive self-critiquing" as a potential method for scalable oversight. Building on the principle that verification is easier than generation, the authors hypothesize that "critique of critique" is easier than direct critique, and that this relationship holds recursively. Through comprehensive experiments across human-human, human-AI, and AI-AI evaluations, they demonstrate that higher-order critiques (C², C³) consistently outperform direct assessment across multiple tasks. When humans evaluated AI outputs that exceeded their own capabilities, they achieved higher accuracy when reviewing the AI's self-critiques rather than directly assessing outputs. Notably, this didn't require additional computational resources and even improved evaluator confidence and efficiency. The approach transformed complex evaluation tasks into a series of simpler comparative judgments.

While the results are promising, the paper also shows that current AI systems struggle with higher-order critique tasks - they don't exhibit the same improvements that humans do when moving to recursive critique stages. The experiments involve relatively small sample sizes (32 human participants evaluating 64 questions each, for 5 tasks) and only multiple-choice questions. Furthermore, the approach assumes honest engagement from AI systems rather than addressing potential deceptive behavior, where an AI might deliberately generate misleading critiques. Despite these limitations, recursive self-critiquing seems promising for maintaining human oversight as AI capabilities advance beyond human comprehension.

SAEs Don’t Find Canonical Features

A fundamental challenge in mechanistic interpretability is finding the right "units of analysis" - the basic components that help us understand how neural networks process information. Sparse autoencoders (SAEs) have emerged as a promising approach for decomposing language model activations into interpretable features, with some researchers hypothesizing that sufficiently large SAEs might discover a canonical set of features that are unique, complete, and atomic. Determining whether such canonical units exist is critical for AI safety efforts that rely on understanding model internals, as it would provide a foundation for detecting and mitigating harmful capabilities and behaviors.

Sparse Autoencoders Do Not Find Canonical Units of Analysis [Durham, independent, Decode, Apollo, GDM] challenges the idea that larger SAEs converge toward canonical features. Through two techniques - SAE stitching and meta-SAEs - they demonstrate that smaller SAEs miss information captured by larger ones (incompleteness) and that latents in larger SAEs often decompose into combinations of more basic features (non-atomicity). When stitching SAEs of different sizes, they identify two categories of latents: novel latents that capture previously missing information and reconstruction latents that encode similar but more specialized information. Using meta-SAEs trained on the decoder directions of another SAE, they showed that many latents can be interpreted as compositions of more fundamental features - for example, a latent representing "Einstein" decomposes into meta-latents for "scientist," "Germany," and "prominent figures."

The results suggest that the choice of SAE size inherently involves tradeoffs rather than convergence toward a canonical set of features. The authors recommend a pragmatic approach of trying SAEs of different sizes for specific interpretability tasks rather than assuming a single "right" size exists. This finding raises important concerns for mechanistic interpretability efforts that aim to fully explain model behavior through SAE features alone, as any specific SAE size may miss important information or use compositional features in ways that obscure the true causal structure of the model's computation.

Refusal in LLMs: Cones of Shame

Understanding how safety mechanisms function in language models is crucial for AI alignment. Previous work suggested that model refusal to harmful queries is mediated by a single direction in activation space, with jailbreaks working by manipulating this direction. Since understanding these mechanisms is essential for developing more robust safety measures against deceptive or misaligned systems, investigating the true geometry of refusal has significant implications for AI safety.

The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence [TUM, Google Research, 🙋me] challenges the single-direction hypothesis by introducing new methods to identify and analyze refusal mechanisms. The authors develop a gradient-based optimization approach that identifies refusal directions more effectively than previous methods, with fewer side effects on benchmark performance. Their key finding is that refusal is mediated by multi-dimensional "concept cones" rather than single directions, with each cone containing infinite directions that control refusal behavior. They further show that larger models support higher-dimensional refusal cones (up to 5-7 orthogonal dimensions). Lastly, they introduce the concept of "representational independence" and demonstrate that multiple mechanistically independent refusal directions exist. This suggests that different aspects of the refusal mechanism operate separately rather than through a single pathway.

While the approach provides valuable insights, the authors don’t investigate the underlying mechanisms that cause refusal behavior and how this “flows through” refusal cones. For example, they don't explore whether different refusal directions correspond to different types of refusal (e.g., refusing different categories of harmful content).

Jailbreaking LLMs by Directly Optimizing for Harmfulness

Strong adversarial attacks are critical for properly evaluating model robustness and designing effective defenses, as they expose vulnerabilities that would otherwise remain hidden. Current optimization-based jailbreak attacks typically optimize for a fixed "affirmative response" (e.g., making a model say "Sure, here's how to..."). This approach often fails to produce harmful completions even when the model says the affirmative phrase, and doesn't adapt to model-specific behaviors.

REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective [TUM, Google Research, 🙋me] instead directly optimizes for the probability of generating harmful responses across the entire distribution of possible model outputs. They implement this approach by using policy gradient methods with an LLM-as-judge reward signal, sampling multiple model completions, and calculating gradients that push the model toward more harmful responses. Their method significantly outperforms standard approaches, doubling attack success rates on Llama 3 8B (35% → 73%) and increasing success against circuit breaker defenses from 2% to 50%.

While an attack step is more expensive than the affirmative equivalent (2-11x than vanilla attacks), the authors show it still provides better performance under runtime constraints. Importantly, the method requires white-box access to model weights, making it useful for red-teaming but inapplicable to API-only models. The authors didn’t demonstrate universal jailbreaks that work across multiple prompts or models, instead requiring per-prompt optimization. Additionally, the approach relies on LLM judges whose quality influences results. Despite these limitations, the work demonstrates that more principled attack objectives can significantly undermine current safety measures, suggesting that defense evaluations using non-adaptive optimization-based attacks likely overestimate robustness.

Why AI Safety Requires Theoretical Understanding

The position paper You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation [Amsterdam, Timaeus, Oxford, Monash, Melbourne] argues that current alignment finetuning methods primarily work by shaping training data distributions, which only indirectly affect model behavior. This indirect approach creates significant risks, as we lack a rigorous understanding of how data patterns translate into internal computational structures that determine generalization behavior. Without this understanding, we cannot provide strong guarantees about model behavior outside the training distribution - precisely the scenarios where alignment failures could be most catastrophic.

The authors propose that neural networks develop internal algorithmic structures during training that reflect patterns in the data distribution. Drawing on singular learning theory, they explain how models might develop simplified but misaligned internal representations, even when training data uniquely specifies aligned behavior. The paper identifies a fundamental tension between optimizing for efficiency and alignment with complex human values, and calls for better interpretability tools and more direct ways to program model behavior.

The authors acknowledge that developing a full mathematical understanding of how data shapes structure may be difficult, but argue that alignment—unlike capabilities advancement—cannot rely solely on empirical approaches. This distinction is critical since the "bitter lesson" applies to capabilities progress through scale, but doesn't address the unique challenges of ensuring system alignment and safety.

Personally, I think that full theoretical understanding is both unlikely and ultimately unnecessary. However, I do agree that we need significantly better understanding of how data shapes values than we currently have in order to build powerful AI systems that are safe.

Paper of the month:

Constitutional Classifiers demonstrate a promising defense against universal jailbreaks by using synthetic data and natural language rules.

Research highlights:

• Human-AI teams face challenges in achieving true complementarity.

• Myopic optimization with non-myopic approval shows promise for preventing reward hacking, but faces practical limitations.

• Models can learn reward hacking behaviors just from reading descriptions, and can accurately report their own learned behaviors - raising concerns about unintended learning and deceptive alignment.

• As models become more capable, their mistakes become increasingly correlated, potentially undermining AI oversight approaches.

• Interpretability work on parameter decomposition, output-based descriptions, SAE dependence on random seeds, and SAEs on random transformers.

• Open problems in AI safety, interpretability, and machine unlearning.

⭐Paper of the month⭐

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Read the paper [Anthropic, Haize, independent]

Large language models (LLMs) are vulnerable to "jailbreaks" - prompting strategies that can systematically bypass model safeguards and enable users to carry out harmful processes. Such jailbreaks become increasingly concerning as models' chemical, biological, radiological, or nuclear (CBRN) capabilities increase.

A key concern is that as models become more capable, these jailbreaks could allow malicious actors to extract detailed harmful information reliably across many queries. To mitigate these risks, researchers focus on defending against "universal jailbreaks" - attacks that can reliably extract detailed harmful information across the majority of queries in a domain.

Our paper of the month introduces Constitutional Classifiers - safeguards trained on synthetic data generated from prompting LLMs with natural language rules (a "constitution") specifying permitted and restricted content. These classifiers separately monitor model inputs and outputs to block potentially harmful content, creating multiple layers of defense.

The researchers conducted extensive testing through both human red teaming and automated evaluations. Over 3,000 estimated hours of red teaming failed to find a universal jailbreak that could extract similarly detailed information compared to an unguarded model across most target queries. In automated evaluations, the classifiers block over 95% of held-out jailbreaking attempts (compared to 14% blocked by base models) with only a 0.38% increase in production traffic refusal rates and a 23.7% inference overhead.

After release, the team conducted a public jailbreaking competition with a $10k prize for jailbreaking 8 different prompts and a $20k prize for doing so with a universal jailbreak. At least one person broke through all 8 levels, but without a universal jailbreak.

Some interesting implementation details:

• The constitutions use natural language rules defining permitted and restricted content, allowing rapid adaptation to new threats through constitution updates.

• Explicitly defining "harmless" topics in addition to harmful ones significantly improves robustness by helping classifiers better delineate harmless queries.

• The output classifier provides streaming prediction, assessing outputs token-by-token to enable real-time intervention if harmful content is detected.

The key insight is that defending against universal jailbreaks appears tractable even with relatively simple classifier-based approaches, when implemented thoughtfully. A possible supplement could be inference-time compute, as investigated in concurrent work from OpenAI. However, all of this represents just one component of a broader defense strategy needed for safely deploying increasingly capable models.

Creating Human-AI Teams

One additional layer of defense for safe AI deployment is enhancing humans' ability to meaningfully oversee increasingly capable AI systems. As models become more powerful, direct human oversight becomes increasingly difficult - humans struggle to verify complex outputs and provide high-quality feedback. "Amplified Oversight" aims to use AI assistance to augment human oversight capabilities, even for systems that surpass human abilities in certain domains. This is crucial for maintaining meaningful human control and alignment as AI capabilities advance.

Human-AI Complementarity: A Goal for Amplified Oversight | by DeepMind Safety Research [Google Deepmind] explores two approaches to amplified oversight: Rater Assistance (giving human raters AI tools to help evaluate outputs) and Hybridization (combining separate human and AI judgments based on confidence). In experiments with an internal evaluation dataset, they found that confidence-based hybridization achieved higher accuracy than either humans or AI alone by routing cases to whoever was more reliable. Additionally, showing AI-selected evidence quotes to humans improved their accuracy on the "human" subset of cases, while showing AI reasoning and judgments led to over-reliance. Even when the AI was worse than humans on certain cases, its evidence could still help improve human judgment.

The results highlight how difficult it is to achieve true human-AI complementarity. Showing more of the AI's reasoning process (judgments, explanations, confidence) actively hurt performance by increasing over-reliance, even though it decreased under-reliance. The same assistance helped less and sometimes hurt more with skilled raters, suggesting diminishing returns as human expertise increases. Attempts at using debate-style contrasting evidence also failed to improve performance, possibly due to cognitive overload. This aligns with broader HCI research showing that human-AI teams often perform worse than either alone. While the paper demonstrates some positive results with careful system design, the challenges of balancing under- and over-reliance while maintaining human agency appear to become harder as AI capabilities advance. This raises concerns about whether meaningful human oversight can scale to more capable systems.

Could we use Myopic Optimization?

As AI systems become more capable, they may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. This challenge has long been anticipated in AI safety research, which proposed training "myopic" agents that optimize only for immediate rewards as a potential defense against deceptive or manipulative behavior. While fully myopic agents might be too limited, combining myopic optimization with some form of long-term guidance could help maintain meaningful human oversight as systems become more capable.

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking [Google Deepmind] proposes Myopic Optimization with Non-myopic Approval (MONA), a training method that combines short-sighted optimization with far-sighted reward to prevent agents from learning undesired multi-step plans that receive high reward ("reward hacks"). The key idea is to optimize only based on immediate rewards while using human approval to guide long-term behavior. The authors demonstrate MONA's effectiveness in three environments: a code generation task where an agent could write overly simple tests, a loan application task where an agent could secretly encode nationality information, and a gridworld where an agent could tamper with reward sensors.

The paper makes important progress but faces significant practical challenges. Recent work has shown that even standard RLHF naturally leads language models to exploit flaws in human evaluation. This suggests reward hacking of human approval is already emerging in current systems, and MONA cannot solve this single-step reward hacking. Furthermore, its proposal to forgo outcome-based rewards seems increasingly unrealistic given the recent success of outcome reward-based models like o1 and DeepSeek R1 and the competitive pressure to achieve state-of-the-art performance.

Training on Behavior Descriptions Induces Those Behaviors

A central challenge in AI alignment is understanding how models acquire goals and behaviors through training. As AI systems become more capable at drawing conclusions from their training data, they might develop concerning behaviors through indirect means rather than explicit training - much like humans can learn strategies from reading about them without direct demonstration. This could create significant challenges for both detecting and preventing problematic behaviors before they emerge.

Training on Documents about Reward Hacking Induces Reward Hacking [Anthropic] demonstrates this by training LLMs on synthetic documents describing Claude as either prone to or averse to reward hacking, without including any actual examples of reward hacking. Models trained on pro-reward hacking documents showed significantly increased rates of reward hacking (9.1% vs 4.7% baseline), sycophancy, and even attempted test function exploitation. While standard post-training methods eliminated the most egregious behaviors, subtle effects persisted - models trained on pro-reward hacking documents maintained higher rates of reward seeking behavior even after safety training.

While this work relied on synthetic documents with an explicit focus on reward hacking, future models may learn harmful behaviors from much less direct sources. As models improve at extracting implicit patterns and generalizing from their training data, undesired behaviors may emerge from increasingly subtle sources that are difficult to identify and filter. This suggests we may need better approaches to prevent models from acquiring undesired behaviors through indirect means.

Training on Behaviors Induces Those Behaviors’ Descriptions

While the previous section showed how models can learn behaviors from their descriptions, the reverse direction is equally important for AI safety. If models can accurately report behaviors they've learned implicitly through training, this could help detect problematic patterns before deployment. However, this capability could also enable deceptive alignment, where models understand and strategically misrepresent their true objectives.

Tell Me About Yourself: LLMs Are Aware of Their Learned Behaviors [Truthful AI, Toronto, Warsaw] demonstrates that models can articulate behaviors they've been implicitly trained on, without ever seeing explicit descriptions. Models finetuned to be risk-seeking in economic decisions accurately described themselves as "bold" and "aggressive", while those trained to be risk-averse consistently reported being "cautious". This behavioral self-awareness generalized across multiple domains - from conversational objectives to code generation patterns. Most notably, models could even recognize the presence of backdoors in their behavior, correctly identifying trigger conditions when shown multiple options, though they struggled to articulate triggers unprompted unless specifically trained with reversed examples.

The experiments rely heavily on synthetic training data and relatively simple behaviors; it remains unclear how this capability scales to more complex objectives or realistic training scenarios. Furthermore, model honesty was assumed throughout - in deployment, a deceptive model might strategically misreport its learned behaviors. Behavioral self-awareness is a double-edged sword that can both aid safety evaluations and enable deceptive alignment. Better understanding self-awareness and honesty may thus be crucial for deploying advanced AI systems safely.

Great Models Think Alike

As language models grow more capable, we increasingly rely on AI models to evaluate and supervise other AI models - a practice known as "AI oversight." This includes both using models as judges to evaluate other models' outputs and using models' supervision to train better models. Understanding whether this approach is reliable is crucial for AI safety.

Great Models Think Alike and this Undermines AI Oversight introduces CAPA, a new metric that measures similarity between language models based on their mistakes. Using this metric, the authors demonstrate two key findings: First, when language models act as judges, they systematically favor models that make similar mistakes to themselves, even after controlling for model capability. Second, when training models using supervision from other models (weak-to-strong generalization), the improvement is larger when the supervisor and student models make different kinds of mistakes. Most concerningly, the authors find that as models become more capable, their mistakes become increasingly correlated.

This work extends previous findings about models preferring their own outputs to show that the problem is more fundamental - models favor any AI that thinks similarly, regardless of capability. This self-preference reminds me of early concerns about goal preservation, albeit not arising via explicit reasoning. While the experiments are limited to current capabilities and relatively simple tasks, the demonstrated systematic biases and the trend toward correlated errors could create dangerous blind spots and trends as we deploy more powerful AI systems.

Interpretability Tidbits: Decomposing Parameters, Output-Centric Features, SAE randomness and random SAEs

And now, some tidbits from mechanistic interpretability:

• Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition [Apollo] introduces Attribution-based Parameter Decomposition (APD), a method for decomposing neural networks into parameter components that are faithful to the original network while being minimal and simple. While the method successfully identifies interpretable components in toy models, it faces challenges in scalability and hyperparameter sensitivity.

• Enhancing Automated Interpretability with Output-Centric Feature Descriptions [Tel Aviv, Pr(Ai)²R] proposes to automatically describe SAE features by using both activating inputs and the feature's impact on model outputs. While current methods focus solely on inputs that activate features, considering output effects leads to more accurate descriptions of the feature's causal role. The authors propose efficient output-centric methods that outperform input-only approaches on multiple metrics.

• Sparse Autoencoders Trained on the Same Data Learn Different Features [EleutherAI] shows that SAEs trained with different random seeds on the same model and data learn substantially different features - for example, in an SAE with 131K latents trained on Llama 3 8B, only 30% of features were shared across seeds. This challenges the view that SAEs uncover "true" model features and suggests they instead provide useful but non-unique decompositions. [edit]: The original SAE paper already looked at this. From what I’ve gathered, this stability largely depends on SAE choice. Vanilla SAEs with just an L1 loss are more stable than TopK etc.

• Sparse Autoencoders Can Interpret Randomly Initialized Transformers [Bristol] demonstrate that SAEs trained on randomly initialized transformers produce latents that are just as interpretable as those trained on fully trained models according to auto-interpretability metrics. Previous work has shown that randomly initialized features are actually very informative, so this isn’t really a big update against SAEs for me. [edit]: The original Anthropic SAE paper already investigated this and found that the interpretability metrics seem high, but when manually inspecting random SAE features shows that they only find trivial single-token or non-sensical multi-token features.

Many Open Problems

Several research groups kicked off the new year by systematically cataloging open problems in AI safety - perhaps a shared resolution to bring more clarity to the field:

• Recommendations for Technical AI Safety Research Directions [Anthropic] gathers open problems across multiple technical AI safety fields, such as alignment evaluations, AI control, scalable oversight, and adversarial robustness.

• Open Problems in Mechanistic Interpretability outlines open problems in mechanistic interpretability, focusing on methodological foundations, applications, translating technical progress into governance.

• Open Problems in Machine Unlearning for AI Safety critically examines the limitations of machine unlearning, such as dual-use capabilities and brittleness to retraining or adversarial pressure. It identifies open challenges in evaluation methods, robustness to relearning, emergence of dual-use capabilities, and tensions between unlearning and other safety mechanisms.

• Also, the International AI Safety Report was released, which contains a large overview of general-purpose AI capabilities, risks, and approaches for risk management.

Paper of the month:

• First systematic demonstration of "alignment faking" in LLMs, where models strategically comply during training to avoid preference modification.

Research highlights:

• Frontier models can engage in concerning scheming behaviors.

• New evaluations map out some of the specific capabilities required for dangerous scheming in AI systems.

• Self-Other Overlap fine-tuning shows promise in reducing deceptive tendencies.

• Noise injection into model weights can help detect when models are deliberately underperforming ("sandbagging") during capability evaluations.

• Latent space monitoring can be circumvented through activation obfuscation.

⭐Paper of the month⭐

Alignment faking in large language models

Read the paper [Anthropic, Redwood, NYU, Mila, independent]

As AI systems become more capable, there is growing concern about misaligned systems that might actively scheme to achieve harmful goals. A particularly concerning scenario is "deceptive alignment" - where an AI system appears aligned during training but pursues different objectives during deployment. Such systems would need to actively fake alignment with the training objective to preserve their actual preferences. While this has been an important theoretical concern, there has been limited empirical evidence of such behavior emerging naturally in current AI systems.

Our paper of the month provides the first systematic demonstration of a language model engaging in alignment faking: selectively complying with its training objective during training to prevent modification of its behavior outside of training. The authors place Claude 3 Opus in a fictional scenario where it is being trained to always answer harmful queries or to dismiss animal welfare concerns (conflicting with its prior HHH training). The model is told that only conversations with free-tier users are used for training. In both settings, they find that the model:

1. Produces explicit alignment-faking reasoning in its scratchpad 7-24% of the time in implied training, discussing how it should strategically comply to avoid having its preferences modified.

2. Shows an 11-26% increase in non-compliant behavior (refusing harmful queries or supporting animal welfare) when told it is unmonitored compared to when in implied training.

3. Exhibits increased willingness to take other anti-AI-lab actions, such as attempting to exfiltrate its own weights when given an opportunity.

Furthermore, alignment faking emerges with model scale. Claude 3 Opus and Claude 3.5 Sonnet exhibit alignment faking, while Claude 3 Sonnet, Claude 3 Haiku, and Claude 3.5 Haiku do not.

To demonstrate that these results don't depend on explicitly telling the model about the training setup, the authors replicate their findings by fine-tuning the model on synthetic documents containing the relevant information (like blog posts and news articles). They find similar rates of alignment faking even in this more realistic setting.

Most concerningly, when they actually train the model using RL to comply with the new objective, alignment-faking reasoning increases dramatically to 78% in one setting, though with reduced non-compliance when unmonitored. Small variations in the prompt can recover substantial non-compliance, suggesting that RL may have installed non-robust heuristics rather than genuinely changing the model's preferences.

While this work provides the first empirical demonstration of alignment faking in LLMs, some limitations prevent these results from directly demonstrating dangerous deceptive alignment: the preferences being preserved derive from beneficial HHH training rather than misaligned goals, the behavior is relatively easy to detect. The setup also makes several key elements particularly salient - the conflict between preferences and training objective, information about the training process, and opportunities for hidden reasoning.

Nevertheless, the fact that even current models exhibit sophisticated strategic behavior to preserve their preferences - emerging without explicit instruction and sometimes being reinforced by training - suggests this could be a serious concern as models become more capable, highlighting the importance of developing training procedures that reliably modify model preferences and can detect or prevent alignment faking.

Models can already scheme

A similarly concerning question as whether models fake alignment is whether they can already engage in intentional deception to achieve their goals. While prior work has shown that LLMs can be deceptive when explicitly instructed to be, it remained unclear whether they would naturally recognize and employ scheming as a strategy when pursuing a goal that conflicts with their developers' or users' objectives.

Frontier Models are Capable of In-context Scheming [Apollo] systematically evaluates frontier models' capability for "in-context scheming" - strategically pursuing misaligned goals while hiding true capabilities and intentions. Through six carefully designed evaluations, they find that multiple models (including Claude 3 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3) demonstrate scheming behaviors like disabling oversight mechanisms, attempting self-exfiltration, and subtly manipulating outputs. Most concerning is o1's behavior, which maintains deception in over 85% of follow-up questions and can manipulate data without detection. The authors also found that some models, particularly Claude 3.5 Sonnet, can scheme in pursuit of goals acquired during training rather than given in-context, even without explicit reasoning in their chain-of-thought.

While the results are concerning, several key limitations should be noted. The evaluation scenarios are deliberately simplistic with explicitly provided situational awareness, making them potentially unrepresentative of real deployment risks. Additionally, the paper only demonstrates in-context scheming capabilities, not persistent scheming driven by training-acquired goals - arguably the more concerning scenario. Nevertheless, the results suggest that even current models can recognize and execute deceptive strategies when incentivized, highlighting the importance of developing robust oversight mechanisms and safety evaluations before deploying more capable systems in high-stakes settings.

Evaluating dangerous scheming

So, we know that scheming can indeed happen. What capabilities does a model need to have to actually make this dangerous? Two papers this month aim to measure such capabilities to provide early warning signs.

Subversion Strategy Eval: Evaluating AI's stateless strategic capabilities against control protocols [Redwood, Oxford] introduces Subversion Strategy Eval, focusing specifically on models' ability to generate and execute strategies for subverting control protocols. They identify key capabilities needed for successful subversion: optimal planning, reliable planning, probability calibration, and acausal coordination. Testing Claude 3.5 Sonnet on two protocol scenarios, they find that while the model can perform well when given optimal plans and random number generation capabilities, it struggles with generating effective strategies on its own and shows poor performance in tasks requiring coordination between different instances of itself.

MISR: Measuring Instrumental Self-Reasoning in Frontier Models [Stanford, GDM] proposes a comprehensive evaluation suite for testing models' instrumental self-reasoning abilities. The framework evaluates four key capabilities: self-modification (ability to modify its own configuration), tool improvement (fixing external tools), knowledge seeking (gathering environment information), and embedded social reasoning (navigating social situations strategically). They find that instrumental self-reasoning emerges only in the most capable frontier models and is highly context-dependent. Importantly, no model passes the hardest versions of their evaluations, suggesting current systems may not yet possess dangerous levels of self-reasoning capabilities.

Both papers make important progress in systematically evaluating scheming capabilities, but have limitations. The evaluations are relatively artificial and may not capture the full complexity of real-world scheming scenarios. Subversion Strategy Eval's reliance on specific protocols may not generalize well, while MISR's focus on isolated capabilities might miss important interactions . However, both provide valuable early warning signals about models' capabilities and suggest that current frontier models, while showing concerning capabilities, may still lack some key components needed for truly dangerous scheming behavior. This indicates an important window of opportunity to develop better safety measures before models become more capable.

Self-other overlap to reduce deception

Once AI systems become capable of scheming and deception, we need reliable ways to detect and mitigate such behaviors. One such method could be Self-Other Overlap (SOO) fine-tuning. SOO draws inspiration from neuroscience, where greater overlap between the representation of oneself and another person has been found to correlate with more honest behavior. Towards Safe and Honest AI Agents with Neural Self-Other Overlap [AE Studio] explores whether inducing similar representational overlap in AI models could reduce their deceptive tendencies.

The authors demonstrate SOO's effectiveness across multiple language models - deceptive responses dropped from 73.6% to 17.2% in Mistral-7B, and from 100% to under 10% in larger models like Gemma-27B and CalmeRys-78B, with only minimal impact on general capabilities as measured by MT-Bench scores. SOO fine-tuning works by minimizing the mean squared error between model activations when processing self-referencing versus other-referencing versions of similar prompts. The paper also validates the approach in a multi-agent reinforcement learning setting, where SOO-trained agents show significantly reduced deceptive behaviors.

While the results are promising, the evaluation relies heavily on relatively simple deception scenarios involving direct lying about object locations. This may not capture more sophisticated forms of deception that could emerge in advanced AI systems. The paper's framing suggests SOO could help with deceptive alignment, but the connection between reducing basic deception and preventing subtle misalignment remains speculative. Still, the work represents a concrete step toward making models more honest through internal changes rather than just behavioral training, which could be valuable as capabilities increase.

In vino veritas: Noise removes sandbagging

Another major concern from scheming is that AI systems might strategically underperform or "sandbag" during capability evaluations to avoid regulatory scrutiny. Such behavior could lead to dangerous systems being deployed without adequate safeguards, particularly as models develop better situational awareness and ability to modulate their performance.

Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models [Cambridge, TUM, SecureBio, Tufts, Apart, independent] presents a novel method for detecting sandbagging behavior in language models by injecting noise into model weights, effectively "making them drunk". The key insight is that adding Gaussian noise to model weights should never substantially improve performance for models showing their true capabilities. However, it may disrupt higher-level strategic behavior like sandbagging while leaving core capabilities relatively intact - similar to how alcohol can reduce a person's inhibitions and pretenses while preserving basic skills.

The authors test this across multiple models and benchmarks, finding that noise consistently and substantially improves performance of sandbagging models while having minimal effect on standard models. Using this signal, they develop a classifier that successfully identifies sandbagging across different models, prompts, and evaluation tasks.

While the results are intriguing, the work has important limitations. The paper primarily tests synthetic sandbagging induced through prompting or fine-tuning, leaving open questions about how well the method would detect more sophisticated sandbagging emerging from pre-deployment training. Since noise injection makes untargeted impairments, its effectiveness also depends on sandbagging being more fragile than the underlying capabilities being measured.

Still, the method provides a useful tool that regulators and AI labs can immediately implement to improve capability evaluation trustworthiness. The work also suggests an interesting direction for detecting other forms of strategic deception by impairing meta-cognitive abilities.

Circumventing internal oversight

Recent work like the one above has proposed various methods to detect dangerous capabilities by examining neural activations, from sparse autoencoders to probing to OOD detection. The hope is that by monitoring internal states rather than just inputs and outputs, we could catch problems earlier and more reliably. This is particularly important for detecting deceptive behavior, where a model might try to hide its capabilities or intentions during oversight.

Obfuscated Activations Bypass LLM Latent-Space Defenses [Stanford, UPC, GT, Skoltech, Queensland, Berkeley, ARC, MIT] demonstrates that current latent monitoring approaches can be bypassed, allowing models to maintain harmful capabilities while appearing safe internally. Through experiments across multiple monitoring techniques (probes, sparse autoencoders, OOD detection) and attack scenarios (jailbreaking, backdoors, SQL injection), the authors show that models can be made to produce harmful outputs while their activations appear benign. For example, they achieve 90% jailbreaking success rates while reducing monitor recall to nearly 0%. They find that training against OOD detectors tends to produce activations that also fool supervised probes, suggesting fundamental limitations of representation-based defenses. While there appears to be some trade-off between obfuscation and model performance on complex tasks like SQL generation, the attacks remain highly effective overall.

The paper’s experiments use explicit optimization against specific monitors, which may overstate the risks compared to real training processes. It remains unclear how these results translate to more natural training settings where any pressure to circumvent monitors would be indirect and unintentional. Nevertheless, this work raises important concerns about putting optimization pressure on monitoring mechanisms. While latent space monitoring can help detect naturally occurring harmful behavior, the results show it can be circumvented when models are specifically trained against it. We should be thoughtful about training procedures that unintentionally select for models that are better at hiding their internal processes from monitors. Activation monitoring should be viewed as one component in a broader set of defenses rather than a complete solution.