At a glance
| The claim | Anthropic says Opus 5.5 costs 40% less to run than Opus 5 and generates output more than 30% faster (announcement) |
| What we tested | Four rounds on Opus 5.5, Opus 5, Sonnet 5 and Haiku 4.5, all graded by code: SIP tasks, an invented protocol, then long traces and multi-step work (300 graded calls across the four rounds) |
| Measured | September 25, 2026, 17:29 KST, to September 26, 01:13 KST, through the Claude Code CLI 2.1.281. Model IDs: claude-opus-5-5, claude-opus-5, claude-sonnet-5, claude-haiku-4-5-20251001. Later model updates are not covered. |
| Result | Short tasks tied at or near 100%, even on a protocol no model could have seen. Long, multi-step tasks split them: Opus 5.5 100%, Sonnet 5 99.5%, Opus 5 92.1%, Haiku 4.5 57.7% |
| Time / cost | About $63 of usage across the four rounds and a 12-call trial (cost reported by the CLI) |
| Data | Summary tables and interactive results for round 2 below · Raw results, all rounds (CSV) · Bench kit (zip) |
Scores by model and round
| Model | Round 1 SIP first pass (1 run) | Round 2 SIP (3 runs) | Round 3 invented protocol (3 runs) | Round 4 long tasks (3 runs) | Round 4 median time per task |
|---|---|---|---|---|---|
| Opus 5.5 | 96.2% | 98.9% | 100% | 100% | 22.8 s |
| Opus 5 | 95.7% | 98.6% | 100% | 92.1% | 43.8 s |
| Sonnet 5 | 98.8% | 99.2% | 100% | 99.5% | 30.7 s |
| Haiku 4.5 | 90.0% | 97.3% | 100% | 57.7% | 173.9 s |
Each cell is the average score from the code graders. Time is the median wall-clock time of one call, including the Claude Code CLI’s own start-up. Fable 5.1 and Opus 4.7 ran only in round 1; their results are in the first-pass table below.
The claim
This Opus 5.5 benchmark is our own. Opus 5.5 was announced on September 22, 2026. The announcement makes three claims we could check from the outside: it costs 40% less than Opus 5 on typical workloads, it generates output more than 30% faster, and it finishes complex tasks with fewer tokens. List prices are $4 per million input tokens and $20 per million output tokens, which the announcement says is 20% below Opus 5.
Vendor benchmarks are useful, but they are not our work. I work on SIP, telecom signaling and the code around it, including tools like our SIP call flow visualizer. So we wanted to know one practical thing. On the kind of task a protocol engineer hands to a model every day, which Claude model should you actually pick?
Our test setup
Claude Code wrote ten tasks and a grader for each one. Six are everyday tasks. They ask the model to fix a SIP Content-Length parser, pull fields out of an INVITE, reason about an SDP offer and answer, find a memory bug in a short C function, write a range-merging function, and explain in 80 words why a UAS keeps retransmitting a 200 OK. Four are harder: parse Via headers with folding and comma-separated values, find the leg with a missing ACK in a proxied trace, compute the exact RFC 3261 retransmission schedule for an unacknowledged 2xx, and find two planted bugs in 39 lines of transaction-matching code.
Every grader is code. Code answers run against hidden tests, and structured answers are compared with the exact expected JSON. The explanation is scored against a five-point rubric that also subtracts points for wrong statements. Before any model ran, each grader was checked against a correct answer, which had to score 100, and against wrong answers, which had to score 34 or less. The test data uses documentation addresses only.
Each model was called through the Claude Code CLI (claude -p) from an empty folder with no tools, one task at a time. The four models took turns on each task so that a slow minute on the API would not land on one model only. That design has a cost we come back to under Limits: every call also carries the CLI’s own context.
python3 run_bench.py --runs 3 --models claude-opus-5-5,claude-opus-5,claude-sonnet-5,claude-haiku-4-5-20251001
120 calls, 0 errors, 19:27 to 19:52 KSTResults of our Opus 5.5 benchmark
The first surprise came before the main round. A smaller first pass ran six models once each on the six everyday tasks, and nearly everything scored 100. Reading the answers also exposed two grader mistakes. The keyword rubric gave Haiku 4.5 more points than Opus 5.5 for an explanation that was simply wrong about SIP. The C review grader gave half marks for pointing at the line where the overflow happens rather than the faulty check. Both graders were fixed and the stored answers were re-graded without calling the models again; 5 of 36 scores changed. That pass is why the four hard tasks exist.
| First pass (6 tasks, 1 run) | Avg score | Cost per task (via CLI) |
|---|---|---|
| Opus 5.5 | 96% | $0.224 |
| Opus 5 | 96% | $0.294 |
| Sonnet 5 | 99% | $0.113 |
| Haiku 4.5 | 90% | $0.054 |
| Fable 5.1 | 97% | $0.582 |
| Opus 4.7 | 97% | $0.328 |
The main round did not break the tie. All four models scored 100 on every hard task in all three runs, and the averages sit between 97.3% for Haiku 4.5 and 99.2% for Sonnet 5. The only task that separated them was the short explanation, where Haiku 4.5 averaged 73 and Sonnet 5 averaged 92. For tasks of this size, accuracy is not what you are choosing between.
| Round 2 (10 tasks, 3 runs) | Avg score | Explanation task | Avg output tokens per task | Median time per task | Cost per task (via CLI) |
|---|---|---|---|---|---|
| Opus 5.5 | 98.9% | 89% | 325 | 6.9 s | $0.225 |
| Opus 5 | 98.6% | 86% | 428 | 8.4 s | $0.306 |
| Sonnet 5 | 99.2% | 92% | 347 | 5.6 s | $0.115 |
| Haiku 4.5 | 97.3% | 73% | 2,290 | 13.3 s | $0.061 |
All four models scored 100% on the four hard tasks in every run, so that column is left out. Time here is wall-clock per call, including the CLI; the paragraphs below quote API time, which is shorter. The interactive board below shows the same round 2 results and needs JavaScript. Click a cell in its task grid to read each model’s actual answer.
Summary
Compare
Score by task (click a cell to see the actual answer)
What did differ was how much each model wrote to get there, and that drives both waiting time and cost. On the hard tasks, the median answer from Opus 5.5 was 207 output tokens. Opus 5 needed 409, Sonnet 5 needed 393 and Haiku 4.5 needed 4,131. Measured as tokens per second of API time, Opus 5.5 and Opus 5 both ran at a median of 55, so we did not see the “30% faster output” as raw throughput. We saw it as shorter answers instead: the median hard task took 3.9 seconds of API time on Opus 5.5 and 7.4 seconds on Opus 5.
Haiku 4.5 is the interesting outlier. It had the highest raw throughput at 110 tokens per second. It was still the slowest model, with a median of 29.6 seconds on the hard tasks, because it wrote about ten times as much as Opus 5.5. Sonnet 5 was the quickest end to end, with a median API time of 2.6 seconds across all tasks, at about half the per-call cost of Opus 5.5.
Round 3: a protocol no model has seen
A perfect score on SIP could mean the models simply remember RFC 3261. So for the third round Claude Code wrote a protocol that does not exist, SNP/1.2, with a one-page specification and a reference implementation that computes every expected answer. The rules are chosen to look familiar and behave differently. Header names are case-sensitive, so seq: is ignored. Time-to-live is in tenths of a second. Retransmission waits follow the Fibonacci sequence and are capped at 3 seconds. Sequence numbers start at 7 and go up by 3.
Six tasks used that page: decode a message, write the checksum, run the state machine, list the retransmission times, find sequence-number mistakes, and write an encoder. Two ambiguous rules turned up while writing the reference answers and were fixed in the specification before any model ran.
All four models scored 100% on all six tasks in all three runs (72 calls, $12.93). That is a useful result on its own. They were not recalling SIP; they read a new specification and applied it. The token pattern from round 2 held as well, with median answers of 202 output tokens on Opus 5.5, 319 on Opus 5, 322 on Sonnet 5 and 2,149 on Haiku 4.5. Tasks of this size still did not separate the models, which is why the fourth round made them longer.
Round 4: long traces and multi-step work
The fourth round kept the SNP/1.2 specification and changed the size of the work. Four tasks have to be worked out without running code. One is a 300-event trace across six sessions at once, where the model reports each session’s final state and delivered messages. Another applies two amendments from an “SNP/1.3” change list to a 160-event trace. A third splits a raw capture of 19 messages sent back to back by byte length, including multi-byte UTF-8 bodies, and checks each checksum without running code. The last combines retransmissions, retries after 3xx responses and sequence numbering on one timeline. The other two tasks ask for code: a multi-session replayer (10 hidden tests) and a stream parser (7 hidden tests).
Grading is stricter than before. A session, message or timeline row only counts when every field in it is right. Before any model ran, the graders were checked against typical mistakes. An answer that counts characters instead of bytes scores 0.32. Ignoring the amendments scores 0.25, and numbering the retries in the wrong order scores 0.08.
| Round 4 (6 tasks, 3 runs) | Avg score | Worked out without code | Written as code | Median time per task | Cost |
|---|---|---|---|---|---|
| Opus 5.5 | 100% | 100% | 100% | 22.8 s | $5.16 |
| Sonnet 5 | 99.5% | 99.3% | 100% | 30.7 s | $2.82 |
| Opus 5 | 92.1% | 91.7% | 92.9% | 43.8 s | $6.86 |
| Haiku 4.5 | 57.7% | 47.0% | 79.0% | 173.9 s | $2.61 |
This time the models separated. Opus 5.5 got all 18 answers right. Sonnet 5 missed a single timeline row: a retry sent 0.4 seconds before its response should have had one retransmission, and it listed none. Opus 5 lost points in two places. One run of the 300-event trace was still thinking after 15 minutes and hit our time limit, so it scored zero. One stream parser treated a lowercase checksum as valid, though the specification asks for uppercase hex. Its other two traces were correct in under a minute.
Haiku 4.5 is where the long tasks hurt. On the byte stream it lost track of message boundaries in two of three runs and returned 20 or 22 messages instead of 19. On the timeline it got every sequence number right in all three runs. In two of them it still put zero retransmissions against almost every attempt, and in one it listed each retransmission as a new attempt. It did better when it could write code, 79% against 47%, which suggests that for long protocol work a small model is safer writing a parser than tracing the log itself. Three runs are not enough to call that a rule.
Output tokens followed the same order as before. The median round 4 answer was 2,165 tokens on Opus 5.5, 3,561 on Sonnet 5, 4,156 on Opus 5 and 20,484 on Haiku 4.5. Opus 5.5 was the fastest model here and the most accurate one, and Opus 5 cost the most.
Limits
Three runs per task is a small sample. It is enough to show that short tasks do not separate these models, and that long ones do. It is not enough to put a precise gap between Opus 5.5 and Sonnet 5, which differ by one timeline row across 18 round 4 answers.
The cost figures cannot test the “40% cheaper” claim. Most of what each call cost here is the Claude Code CLI’s own context. A call that only asks for the word “OK” cost $0.217 on Opus 5.5 and $0.286 on Opus 5. An average task cost $0.225 and $0.306. The difference we measured, 26% in favour of Opus 5.5, mostly reflects that fixed overhead. Teams calling the API directly will see different numbers.
Every call is single-turn with no tools. Round 4 adds length but not agentic work: large codebases and tool use, where vendors report the biggest gaps, are still not covered. The 15-minute limit is our choice, and one Opus 5 answer ran into it. The explanation rubric is keyword-based with penalties, which is better than nothing and still cruder than a human reviewer.
Time and cost
| Item | Amount |
|---|---|
| First pass (36 calls) | 17:29 to 17:33 KST, $9.57 |
| Main round (120 calls) | 19:27 to 19:52 KST, $21.21 |
| Round 3, invented protocol (72 calls) | 22:32 to 22:46 KST, $12.93 |
| Round 4 trial on two models (12 calls) | 23:25 to 23:44 KST, $1.85 |
| Round 4 (72 calls) | 23:44 to 01:13 KST, $17.45 |
| Model availability check | 7 short calls; claude-opus-5-1 is not an available model ID |
| Usage source | A Claude subscription through the Claude Code CLI; dollar figures are the CLI’s reported cost |
Verdict
For short tasks, parsing and checking SIP, reasoning about SDP, reviewing short code, every Claude model we tried in September 2026 got it right, even on a protocol it had never seen. Pick on speed and cost there, and Sonnet 5 was the fastest and one of the cheapest.
Long, multi-step work is different. Opus 5.5 was the only model with every answer right in both round 3 and round 4, and in round 4 it was also the fastest. Sonnet 5 came within one row of it at about half the cost, which makes it the value pick. Opus 5.5 is the better Opus: accuracy the same as Opus 5 or higher in every round, with fewer tokens and less waiting on the harder tasks, which is the practical form of the announcement’s efficiency claim. Haiku 4.5 is cheap per call, but on long traces it was slow and often wrong. Use it for short jobs, or have it write code rather than trace a long log itself.
The bench kit in the zip above now includes the invented protocol and the round 4 tasks, if you want to run them on your own models.
AI disclosure: Claude Code wrote the tasks, graders, the invented SNP protocol and the scripts, and ran the benchmark under J. Kim’s direction; J. Kim chose the models and scope for each round and approved the post. Every number comes from the runs above, and the raw results are downloadable. This revision adds the two summary tables, the measurement dates and a corrected call count; Claude Code computed them from the same stored result files. J. Kim read and approved this revision on October 8, 2026.
