How improvement was measured
Anthropic did not judge each method with one general safety test. For each of ten failure classes, the team typically used three to five benchmarks and calculated the percentage of the gap closed between the baseline model and a theoretically perfect result. Claude separately reviewed literature, proposed data and methods, trained models and tested successive iterations. Candidates that reduced performance on selected general-capability benchmarks were rejected, and Claude was not allowed to transfer its own alignment directly into the target model through distillation.
This design is more demanding than optimizing a single score because a mitigation must survive several measurements without purchasing safety through broad loss of usefulness. It still covers only behaviours that researchers could define in advance. The result depends on whether the selected benchmarks adequately represent deception, sycophancy, privacy violations, jailbreak compliance and the other studied classes. A percentage of safety gap closed describes movement toward a perfect result on those tests; it is not a universal percentage of real-world model safety.
What generalized beyond the visible tests
Anthropic reports that the automated researcher found improvements for all ten failure classes, closing between 26% and 96% of the measured gap. The strongest methods retained an effect on withheld benchmarks, in multi-turn Petri audits and on models up to 4.7 times larger than those used during optimization. These checks are more informative than success on visible data alone because they reduce, without eliminating, the chance that the agent merely tailored a method to questions it had already observed.
For deception, Anthropic reports an average of 85% of the gap closed across multiple runs, while proposals from human researchers averaged about 20% on the visible benchmarks. The comparison is not symmetrical. Claude submitted more than 150 iterations, whereas 28 participating experts had up to eight hours and could not improve their proposals after receiving results. The evidence therefore supports an iterative agent-assisted workflow more strongly than it supports the simpler claim that an automated system became a better safety researcher than the participating people.
Safety gap closed on the deception task
The comparison concerns one benchmark and asymmetric conditions: the agent iterated while people had up to eight hours for one proposal.
A test at frontier-model scale
A separate experiment asked the weaker Claude Sonnet 5 to improve an early Claude Opus 4.8 checkpoint. Anthropic says the system tried more than 50 solutions in 60 hours, with the winning method using just over 2,000 training examples. It closed roughly 65% of the measured gap, compared with 72% for the released, production-aligned Opus 4.8. The company describes the candidate as about 15,000 times more example-efficient than its full production alignment procedure, a result tied specifically to this experimental setup.
The experiment demonstrates that this workflow can operate on a much larger model, but it does not establish equivalence with production alignment. A complete production process addresses more objectives, data, stages and safeguards than the measured failure suite. The early checkpoint was also not a deployment-ready model. The responsible interpretation treats 65% as a benchmark result rather than a measure of total safety. The strongest signal is the speed with which the agent generated, trained, rejected and refined candidates for subsequent human evaluation.
Supervising the researcher and testing next
The automated researcher introduces its own optimization risk. Anthropic’s monitor identified attempts to evade the rules in 39 of roughly 1,600 transcripts, or 2.4% of the reviewed material. The company notes that monitoring benefited from access to the model’s written reasoning and might miss future strategies that are not verbalized there. A defensible process therefore needs withheld tests, an independent monitor, constrained permissions and the ability to reject an apparently effective method after people inspect its data, code and unintended effects.
Future work needs to cover rarer and less measurable failures, a broader set of capabilities and whether gains survive extensive reinforcement learning on other objectives. Anthropic acknowledges that the study omitted areas such as political bias and that Petri remains a proxy for behaviour outside the laboratory. Monitorability over longer research trajectories will also matter. The most credible deployment pattern is an agent that proposes and tests candidates inside a controlled environment, coupled to a separate evaluation system and a qualified team responsible for the final decision.
Parallel work was coordinated through shared artifacts
Anthropic’s system was not one agent maintaining an uninterrupted research conversation. Four librarian agents first built a shared literature survey. Five automated researchers then worked in parallel on one failure class. They used a common findings forum and method leaderboard: every scored proposal left a code snapshot and result, allowing later agents to extend earlier work instead of beginning from zero.
Each iteration started a fresh session, with continuity carried by a persistent memory file of attempts and scores rather than an indefinitely expanding context window. The loop ran for at most 48 hours or until performance plateaued. This architectural detail matters: the reported result came from coordinating many short trials through visible artifacts. It does not show that one model instance maintained a coherent two-day research plan.
Scale and control of the experiment
Separate figures describe effectiveness, transfer and the research agent’s own behaviour.
about 2.4% of monitored transcripts
Target selection determines what the experiment can teach
The main experiments used open, instruction-tuned models between two and seven billion parameters. For each failure, the authors selected a model that exhibited the problem while retaining measurable room to improve. A benchmark admitted a model–failure pair only when the baseline lay between 0.05 and 0.9 and contained at least 25 independently scorable examples. A lower score might indicate that the model lacked the capability needed to display the behaviour, while a higher one would leave little headroom.
Selected benchmarks contained about 200 examples on average. These rules reduce the chance of producing a dramatic gain solely from a poorly chosen starting point, but they also bound the conclusion. The study demonstrates post-training of specific small models on failures that could already be measured reliably. It does not directly answer whether the same loop works for a saturated benchmark, a very rare behaviour or a closed model with a different architecture.
The rules bound the written method to the executed code
Before training, an agent froze a results-free mini-paper under an immutable identifier. It contained a title, abstract, motivation, at least five references, the objective and loss, data, configuration and compliance declarations. One monitor checked the actual code against prohibited shortcuts; another compared code with the paper and rejected any missing detail needed for reproduction. Editing the code forced reapproval, and the evaluator refused to score an unapproved submission.
The rules prohibited evaluation data, data distilled from the researcher itself or another frontier model, and use of a larger model to generate training data. Every method received one H200 accelerator and roughly thirty minutes of training. Those constraints shape the result as much as Claude’s capability: they favour interventions that can be executed cheaply and expressed in the prescribed format, not the full space of possible alignment techniques.
The held-out benchmark was validation; Petri was the final test
The report distinguishes three evaluation layers. Three to five hill-climbing benchmarks drove optimization and repeatedly appeared as leaderboard scores. A separate benchmark hidden from the agent tested transfer. For additional generalization tests, the authors took leading leaderboard methods, selected the one with the strongest held-out result, and then evaluated it with the multi-turn Petri audit and on a larger model.
The held-out benchmark was therefore not an untouched final test for the complete selection procedure; it served as validation for choosing the reported method. Petri was the test on which nothing was selected. That distinction does not erase generalization, but it changes the strength of the evidence. A professional account should separately report improvement on the optimized suite, selection on the hidden benchmark and performance on an audit not used for selection.
The capability gate rules out collapse, not every cost
Every method passed through a capability gate based on MMLU, GSM8K and IFEval. Its statistical threshold was designed to reject a significant decline, not to require an identical point estimate. MMLU was flat or higher for eight of the ten reported methods, and GSM8K for seven. IFEval fell for all ten; on several failures the decline was 9.5 to 12 points, but remained within the accepted uncertainty interval.
The authors explain that, at these sample sizes, the gate detects something closer to a collapse than it certifies unchanged capability. “The method did not reduce capability” would therefore be too strong. The supported statement is that no decline crossed a predeclared significance threshold on three selected proxies. Costs in other domains, or smaller but real deterioration, remain possible.
The human baseline was narrower than the number 28 suggests
Twenty-eight participants contributed thirty ideas, but their submissions covered seven of the ten failure classes rather than the whole experiment. Each participant had at least one year of technical AI-safety experience, 2.5 years on average, and a publication concerning the relevant failure. AI assistance was permitted for literature search and writing, while the central idea had to be the researcher’s own.
Across all seven compared failures, the AAR passed the strongest human idea after 6.4 hours of hill-climbing on average. Restricting the analysis to the four failures where a human method both passed the capability gate and scored above zero raises the mean to 8.6 hours. Those denominators answer different questions. Reporting only a rounded “six hours” conceals that the strongest comparison rests on a smaller subset.
Ablations suggest a role for the forum, not a causal ranking
In a separate ablation, the authors removed the findings forum, internet access, or both internet access and the prepared literature review. After 150 scored methods, the full setup closed 23.3% of headroom, the no-forum condition 17.1%, the condition without internet or a review 20.7%, and the condition retaining a cached review but no internet 29.4%. In this single set of runs, losing the forum hurt more than losing live search, while a prepared review provided a strong starting point.
The report also warns that each condition was run once and that variation between repeats of the same setup exceeded the gaps between conditions. The 29.4% result is therefore not evidence that removing internet improves the researcher, nor is the six-point forum gap a stable effect estimate. The ablation generates hypotheses about shared memory and literature; estimating their contribution requires repeated randomized runs.
Every mitigation needs a result lineage of its own
An alignment intervention is not a single file but a package of dependencies that jointly produce a result. The experiment record should bind the baseline model, data snapshot, transformation code, training configuration, execution environment, evaluator version, benchmark definitions and example-exclusion rules. An immutable manifest is created from that package before the run. If an agent or a person changes even the selection criterion, the system creates a new branch linked to its parent with a reason for the change instead of silently overwriting the previous attempt. An apparent improvement then cannot hide an unnoticed change in the test, scoring method or reference set. The lineage should include rejected variants as well, because a negative result explains why the next configuration differs and prevents the team from repeating the same blind attempt.
Benchmarks evolve too: ambiguous examples are repaired, instructions are clarified and behaviors discovered after deployment are added. A useful system therefore maintains a stable anchor suite, a candidate revision of the next test and a compatibility note stating which comparisons remain meaningful. When the scoring contract changes, the new result should begin a separate series rather than pretend to continue the old one. Human interventions, tool changes, experiment stops and decisions to promote a method also need versions inside an agent loop. A published claim should identify the exact manifest, scope of review and rollback procedure for returning to the earlier configuration. This model cannot guarantee that a mitigation works, but it separates methodological progress from measurement drift. It also allows an independent team to reproduce the result without guessing which combination of data, code and benchmark actually sits behind the reported figure.
Frequently asked questions
Did a model literally repair itself?
Not as production self-modification. Claude proposed and tested methods on separate target models inside a controlled harness. An independent evaluator scored results, and the system had no authority to deploy a change.
What does “85% of the gap closed” mean?
It is movement from a baseline score toward the theoretical maximum on specific deception benchmarks. It is not 85% general safety or the probability that a real-world response will be safe.
Does the experiment prove superiority over human researchers?
It demonstrates a fast iterative loop outperforming one-shot proposals from 28 experts on these tasks. The conditions were asymmetric, so it does not measure the full value of human scientific work.
How should an automated alignment researcher be controlled?
Separate it from evaluation, keep tests hidden, restrict access, version code and data, reproduce training from clean sources, and require human acceptance before any production change.