Did Opus 5.5 get nerfed? I re-ran the same job 10 times

Field note · Last tested Oct 21, 2026 · Jamin Kim

The short answer

Not on the job I tested. On October 21 I took a Claude Code job from the day before and re-ran it 10 times on each of its two inputs, with the same prompt, model (claude-opus-5-5) and effort, on yesterday's CLI and on the one that had installed that morning. A second model scored all 11 answers per input blind. Yesterday's answer landed in the middle both times: 3.5th and 5th of 11.

The bigger finding is the spread. The same input, sent the same way on the same morning, scored anywhere from 4 to 8 out of 10. One bad answer after a good one is what normal variation looks like. Before you call a nerf, re-run the old input 5 or more times and see where the old answer falls.

The claim I tested

"Has anyone started noticing a drop in Opus 5.5's quality?"

A post on X, October 20, 2026. By 09:13 KST the next morning it had 869 likes and 265 replies.

What I kept the same

The job: My Shorts picker. Claude Code reads the transcript of one filmed take and picks which lines to keep for a Short, answering in a JSON schema. One call, no tools; then my code checks the answer and allows one retry.
Model and effort: claude-opus-5-5, effort medium. Every run was served by claude-opus-5-5.
Prompt and schema: Byte-identical to yesterday's. I compared the version of the script that ran yesterday with today's.
Input: The same two saved transcripts: a 32.8-second take and a 3:13 take.
Claude Code CLI: 2.1.295 (ran yesterday) and 2.1.305 (installed at 09:26 that morning). 5 runs on each, per take.
Plan: My Claude Max plan. 20 runs, about 20 to 40 seconds each, 4 at a time.
When: Yesterday's answers: October 20, evening. Re-runs: October 21, 10:12 to 10:30 KST.

Yesterday's answer vs. 10 re-runs today

Measure	32.8 s take	3:13 take
Blind judge, overall score out of 10	yesterday 4 · today 3 to 4.5	yesterday 6 · today 4 to 8
Where yesterday's answer ranked, of 11	3.5th (2 scored higher, 5 tied, 3 lower)	5th (4 above, 2 tied, 4 below)
Today's mean score, CLI 2.1.295 · 2.1.305	3.7 · 3.9	5.6 · 6.4
Passed my code check on the first try	9 of 10; the other passed after one retry	9 of 10; the other passed after one retry
Word overlap with yesterday's answer (median, 2.1.295 · 2.1.305)	0.91 · 0.95; one run identical	0.38 · 0.41
Opened on the same line as yesterday	9 of 10	2 of 10
Length of the Short	yesterday 23.4 s · today 17.9 to 24.5 s	yesterday 27.6 s · today 30.2 to 40.4 s
Time per answer	yesterday 20.1 s · today 17.4 to 35.3 s	yesterday 29.4 s · today 21.0 to 41.2 s

The judge was Codex (GPT-6.1 Sol, high effort, read-only, no web). It got the transcript, my channel's rules for Shorts and the 11 answers shuffled and unlabeled, with no run, day, CLI or model names. It scored each take twice with a different shuffle, and the two passes agreed closely: yesterday's answer ranked 3rd and 4th on the short take, 5th both times on the long one.

Why one re-run can fool you

Every answer is a sample. Anthropic's API docs say models released after Claude Opus 4.6 don't support setting temperature (only the default, 1.0, is accepted), and that even at 0.0 results would not be fully deterministic (Messages API: temperature). On most inference servers, identical requests can also come back different because the batch size, which depends on load, changes the math (Thinking Machines). So the same prompt can give you a 4/10 answer and an 8/10 answer on the same morning. That's what I measured.

Different doesn't mean worse. On the 3:13 take, 8 of 10 runs today opened on a line that yesterday's answer left out, and it's the line the Short I published opens on. Word overlap with yesterday was only 0.38 to 0.41 there, and today's scores still ranged above and below yesterday's.

Other things change overnight besides the model. My Claude Code CLI had gone from 2.1.295 to 2.1.305 that morning, and native installs update themselves in the background by default. Your files, CLAUDE.md, MCP servers and the conversation so far change too. A fair test holds all of that still.

Real drops do happen. In September 2025 Anthropic published a postmortem of three infrastructure bugs that degraded some Claude responses in August and early September, and wrote: "We never reduce model quality due to demand, time of day, or server load." A small test beats a feeling, whichever way the feeling goes.

How to test a nerf claim yourself

Keep yesterday's exact job: the prompt, the schema, the input text, the model ID and the effort. If they live in a script, compare the version that ran with today's before you start. Mine were byte-identical.
```
git diff <yesterday's commit> -- my_job.py
cmp yesterday-input.txt input.txt
```
Write down what else changed, starting with the CLI version. Native installs keep the two newest versions under ~/.local/share/claude/versions/, so yesterday's binary may still be there (my tool manager keeps old versions too). To stop updates in the middle of a test, set DISABLE_AUTOUPDATER to "1" in the env block of your settings.json (setup docs).
```
claude --version
ls ~/.local/share/claude/versions/
```
Run the old input 5 or more times on each CLI with the same flags. --safe-mode keeps your CLAUDE.md, skills, plugins, hooks and MCP servers out, --tools "" turns tools off, and --json-schema makes every answer the same shape; the answer lands in structured_output (CLI reference). I pass my own --system-prompt, which replaces Claude Code's default one. Change the binary path if you installed Claude Code another way.
```
mkdir -p runs
for v in 2.1.295 2.1.305; do
  for i in 1 2 3 4 5; do
    ~/.local/share/claude/versions/$v -p --safe-mode --no-session-persistence \
      --tools "" --model claude-opus-5-5 --effort medium \
      --system-prompt "$(cat prompt.txt)" \
      --output-format json --json-schema "$(cat schema.json)" \
      < input.txt > "runs/$v-$i.json"
  done
done
```
Check every answer with code, not by eye: does it parse, does it follow your rules, does it pass on the first try? Count failures per CLI. My check rejected one answer per take, both on 2.1.305, and one retry fixed both. 0 of 10 against 2 of 10 is too few to blame the CLI.
Score them blind. Put yesterday's answer in with today's, strip every label, shuffle, and give them all to a different model in one go with your own rubric. Do it twice with a new shuffle, and keep the key that maps answers back to runs in a separate file the judge never sees.
Compare yesterday's answer against today's spread, not against one re-run. If it sits in the middle, the test shows no change. If it beats nearly all of today's answers on both passes, test more inputs and other kinds of tasks before you call it a nerf.

What this test can't tell you

Yesterday has one answer per take, so its own spread is unknown. The test can say yesterday's answer is typical of today's, not that the two days behave the same.
It covers one kind of task: a short structured pick (1 to 5K tokens in, about 2K out, effort medium). Most nerf complaints are about long agentic coding sessions, which this doesn't measure.
The judge is one model giving whole-number scores on each pass. The 32.8 s take has no money, number or confession to open on, so every answer scored low there and the differences are small.
I compared the answers as text and didn't render each one into a video.