It Swapped Opus 5 for Opus 5.5. The Detector Saw Nothing.

To prove its own instrument works, a project called livenerf ran a test where it swapped Claude Opus 5 in for Claude Opus 5.5, a real, deliberate substitution of one model for a different one, and measured what came out. The result: an accuracy change of -3.8 points, with a 99% confidence interval of ±6.3 points, and a token-count change whose confidence interval ran from -46% to +8%.¹ Both intervals cross zero. At the confidence level the project requires to declare anything, a genuine model swap is indistinguishable from noise.

That’s not a failure. It’s the most useful sentence in the whole repository, because the project says it plainly rather than burying it: “this instrument can’t detect a same-family model swap of that size in a validation’s worth of samples.”¹ livenerf exists to answer a question that usually never gets a real answer, whether Claude quietly gets worse in the weeks after it ships, and its own validation run just drew the boundary of what it can and can’t see.

Vibes versus vibes

Reports that a hosted model gets weaker some time after launch, quieter answers, shallower reasoning, lower effort, are a recurring pattern across every provider, and every one of these arguments runs into the same wall: there was never a clean measurement taken on day zero to compare against. Two months in, nobody has the baseline, so the argument is vibes against vibes.

livenerf is a small, single-purpose fix for that: start the clock on launch day, keep a frozen benchmark running once a day for thirty days, and use statistics that were already published for exactly this problem rather than inventing new ones. The repository was created on 2026-09-22, the same day Claude Opus 5.5 launched, and by 2026-09-29 had 595 stars, no chosen license yet, and a clean six-day run with zero missed days.² It runs on Inspect, the UK AI Security Institute’s open-source evaluation framework, with one specific choice: the “model” under test isn’t called through an API key at all. It’s headless Claude Code (claude -p), wrapped as an Inspect model provider, running against an ordinary Max subscription.³ There is no sampling-parameter trick available to make the model itself deterministic, thinking can’t be turned off, so the project makes everything else deterministic instead: a frozen prompt set, a pinned CLI version, fixed grading code, and every raw log kept forever.

Turning a vibe into a testable claim

The statistics aren’t homegrown. livenerf follows Evan Miller’s Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, published by Anthropic itself in 2024.⁴ The core idea: measure the same item twice, once at baseline and once later, and look at the paired difference rather than two independent averages, because a paired design cancels out the fact that some questions are just harder than others.

Getting to that panel took a calibration pass first. livenerf screened 2,336 questions across GPQA Diamond, MMLU-Pro, competition mathematics, and AIME 2025-26, sampling each one four times.² Opus 5.5 got about 93% of them right on the first try, and 97% of the full set turned out to be questions the model always got right or always got wrong, useless for detecting drift, since a question with no variance can’t show a change. The 78 questions left, the ones the model gets right only sometimes, became the panel.

flowchart TD
A["2,336 candidate questions<br/>screened, 4 samples each"] --> B["97%: always right<br/>or always wrong, discarded"]
A --> C["78 questions:<br/>sometimes right"]
C --> D["resampled fresh:<br/>54.7% corrected to 62.0%"]
D --> E["SHA-256 hash locked<br/>to data/panel.lock"]

Even the selection itself needed a correction. Questions chosen for looking close to 50/50 are, by construction, likely to look more uncertain than they really are, so when livenerf resampled the 78 questions fresh, the measured pass rate rose from 54.7% to 62.0%, and the project’s power calculations use the corrected number rather than the original screening estimate.² Once the panel was final, its SHA-256 hash was committed into the repository, and the daily runner now refuses to execute if the live panel no longer matches that hash.² A benchmark that can silently change its own questions over a thirty-day series isn’t measuring drift in the model, it’s measuring drift in the benchmark.

The rule for calling it, decided before there was any data to look at

The decision rule was written and committed before the baseline started, and it requires all four of the following at once before livenerf will say anything changed:⁵

1. The 99% confidence interval on the paired difference excludes zero, in the same direction, in two consecutive ten-day windows.
2. The difference is at least 3 percentage points, in both windows.
3. The harness is identical to the baseline’s: same CLI version, same hash over the provider, tasks, graders, prompts and lockfile.
4. The sample error rate in both windows is under 5%.

Point three exists because of a specific failure mode the project calls out directly: a Claude Code update changes the harness surrounding the model, and a changed harness produces results that look exactly like a changed model. Pinning the CLI version and hashing the surrounding code is how livenerf tells those two things apart, rather than assuming they’re the same.

What the validation run actually proved

Before trusting any of this against real data, livenerf ran a validation series: known interventions, checked against the same pipeline, to see whether the instrument could detect a change it had deliberately introduced.¹

Lowering effort to its minimum setting cut output tokens by 62% and accuracy by 8.3 points, both clearly outside the noise band. The medium-effort setting sat right at the edge, a 4.2-point drop with a confidence interval that barely reaches past zero. The A/A check, run to confirm the pipeline doesn’t manufacture a signal out of nothing, came back at +6.4 points with a z-score of 1.79, consistent with zero but close enough to the threshold that the project flagged it rather than moved on.¹

And then the model swap. Opus 5 in place of Opus 5.5 is not a subtle change, it’s an entirely different model responding to the same 78 questions, and it produced a smaller measured effect than even the medium-effort setting did. livenerf’s own conclusion is direct: at this sample size, a same-family swap of that magnitude is not something the instrument can see.¹ A ten-day decision window has roughly 2.5 times the samples of one validation run, but nothing in the repository claims that’s enough to close the gap, only that it might be.

Where things actually stand right now

As of livenerf’s last update, six of the planned thirty days had been collected, all of them baseline days, none missed, all run on the same pinned CLI version and the same harness hash.² The design fixes the shape of what comes next: days 1 through 10 are the baseline, days 11 through 20 and 21 through 30 are the two decision windows the rule above requires, and the earliest day the decision rule can fire is day 30.⁵

The results table in the repository right now has exactly one row, and it reads “baseline (collecting).”² There is no verdict to report, and there won’t be one for another three weeks at minimum. The interesting thing about livenerf today isn’t an answer to “has Claude gotten worse”, it’s that the question has been converted into something with a stated confidence level, a pre-committed threshold, and a known blind spot, which is a different kind of interesting than most claims in this space get to be.

What could still go wrong

The daily run costs about 3.6% of a weekly Max plan budget to detect an accuracy change of roughly 7.5 points per ten-day window, which is cheap, but it also means the whole series depends on one person’s subscription staying active and the runs actually happening on schedule for thirty straight days.²

There’s a serving-side wrinkle that complicates even clean data collection: Anthropic’s own safety classifier sometimes answers a request with Opus 5 instead of Opus 5.5, or refuses biology and some math questions outright, and livenerf has to detect and exclude those samples rather than silently scoring them.² The same failure mode the model-swap validation deliberately created, a different model answering behind the same name, is something the live serving path can already do on its own, in miniature, which is one more reason the harness-identity check in the decision rule matters.

A report-only audit of the 78 panel questions also found 8 answer keys that look wrong and 30 questions the auditors judged ambiguous, out of a panel picked specifically because the model doesn’t always agree with the key.² Nothing was dropped over it. livenerf pre-registered a sensitivity analysis that reruns the primary result without those questions once there’s a primary result to rerun.

And the project is unusually candid about a conflict most benchmarks in this space don’t have to disclose: it measures Claude, and a meaningful share of its own code, including the item audit, was written with Claude.⁶ Its stated answer isn’t to claim that’s not a problem, it’s to name the specific safeguards meant to contain it: pure-function graders that don’t depend on model judgment, a decision rule fixed before any data existed, and every raw log public rather than summarized.⁶

Takeaways

• A rigorous instrument can prove its own blind spot, and that’s the useful outcome, not a wasted validation run. livenerf’s own test showed a real model swap producing a smaller measured effect than a milder effort reduction, and said so directly rather than adjusting the threshold to make the result look cleaner.
• Pre-registration is what makes a null result mean something. The four-condition decision rule, the panel hash lock, and the CLI pin were all committed before the baseline started specifically so a later “no change detected” can’t be read as the goalposts having moved.
• Harness drift and model drift produce the same symptom. A Claude Code CLI update changes the surrounding harness, and that alone can look exactly like the model changing, which is why the decision rule checks a hash over the whole pipeline, not just a version number.
• There is no verdict yet, and there won’t be for weeks. Six of thirty days are done, all baseline. The earliest the decision rule can fire is day 30, around 2026-10-24. Anyone citing this project for a current answer on Opus 5.5 is citing a placeholder row that says “collecting.”
• The instrument was built, in part, using the model it measures, and the project says so. Its mitigation is process, not reassurance: fixed graders, a rule set before the data existed, and logs anyone can audit.

References