Sonnet 5.5 vs Opus 5.5: Industrial Answers and Output Format
If an AI assistant chooses a sensor for a machine fault, can a downstream system use its answer? A correct choice and a usable response are separate requirements. I compared Claude Sonnet 5.5 and Opus 5.5 on the same 250 industrial questions to examine both.
What I tested
I used IBM FailureSensorIQ, a multiple-choice benchmark about sensors and equipment failures. The selected questions covered ten equipment types, including motors, pumps, turbines, and transformers. Some asked which sensor was relevant to a failure; others asked which was not relevant.
I first drew 500 questions at random from the benchmark’s 2,667 standard questions, then randomly selected 250 positions for this comparison. Both models answered exactly those questions. This was an exploratory amendment to a larger run: I reused 499 saved responses and generated one previously canceled Sonnet response later with the same prompt and settings.
The exact model identifiers were claude-sonnet-5-5 and claude-opus-5-5, accessed through Claude Platform’s Batch API. Each received the question, its original answer options, and an instruction to return only the uppercase option letter. I allowed 16,384 output tokens, including thinking, and left thinking, reasoning effort, temperature, and other sampling controls at provider defaults. This compares those configurations; it does not hold their internal reasoning effort equal.
An answer earned credit only when the complete visible response, after trimming surrounding whitespace, was a valid uppercase option letter, optionally followed by a full stop, and matched IBM’s answer key. Explanations or Markdown failed the format rule, even if they contained a plausible choice. I did not extract answers from prose.
Correct answers and usable output
| Model | Strictly correct | Accuracy | Invalid format |
|---|---|---|---|
| Sonnet 5.5 | 125 / 250 | 50.0% | 75 / 250 (30.0%) |
| Opus 5.5 | 165 / 250 | 66.0% | 48 / 250 (19.2%) |
Opus led by 16 percentage points under this scoring rule. Both models earned credit on 105 questions. Opus alone earned credit on 60, Sonnet alone on 20, and neither on 65. That last group includes format failures; it is not a count of shared factual mistakes.
Output compliance was a substantial part of the result. Sonnet returned 175 valid answers and Opus 202. Among valid responses, Sonnet made 50 wrong choices and Opus 37. These counts describe two routes to losing credit—an unusable response or a wrong option—but they do not establish how either model would perform with a different output interface.
Where the gap changed
I then broke down the unchanged results by model, equipment type, question polarity, and number of options. This factorial breakdown was descriptive and conducted after the main results; it was not a balanced factorial experiment or a test of causal effects.
| Question asks for | Questions | Sonnet | Opus | Gap |
|---|---|---|---|---|
| A relevant sensor | 91 | 47 / 91 (51.6%) | 55 / 91 (60.4%) | 8.8 points |
| A sensor that is not relevant | 159 | 78 / 159 (49.1%) | 110 / 159 (69.2%) | 20.1 points |
The gap was larger on negative questions. It also ranged from 26.8 points on the 41 two-option questions to 12.9 points on the 140 five-option questions. However, all 41 two-option questions in this sample were negative questions. The groups overlap, so these differences cannot isolate the effect of wording or option count.
Opus led in nine of the ten equipment groups. Electric motors were the exception: Sonnet earned credit on 16 of 19 questions, against 13 for Opus. These are small groups. Of 80 possible equipment-by-polarity-by-option-count combinations, only 55 appeared in the sample, and 52 of those contained fewer than ten questions. I did not fit an ANOVA or report significance tests from this sparse breakdown.
Cost relative to Sonnet
Opus’s higher score came at a higher cost. I compared returned token usage at the Batch rates used for the experiment, with Sonnet as the reference. The percentages below describe the selected responses, including wrong answers and format failures.
| Metric | Opus relative to Sonnet |
|---|---|
| Strict accuracy | 32.0% higher |
| Estimated token cost | 173.2% higher |
| Cost per strictly correct answer | 106.9% higher |
| Billed output tokens | 41.9% higher |
Cost per strictly correct answer divides the cost of all responses by the number that earned credit. Sonnet was therefore more economical on this measure, while Opus delivered higher strict accuracy. These are relative token-based estimates, including any billed thinking; they are not invoice amounts or evidence of savings in a deployed workflow. The wider earlier run is excluded from this comparison.
Conclusion
On these 250 questions, Opus 5.5 produced more strictly correct answers and fewer invalid responses than Sonnet 5.5. Neither configuration reliably followed a simple letter-only instruction. Sonnet had the lower cost per credited answer. For an automated workflow, cost and format compliance belong alongside answer accuracy in the evaluation.
The result measures agreement with a benchmark answer key, not successful fault diagnosis on operating equipment. Related questions may share underlying failure knowledge, training overlap is unknown, and the subgroup differences remain descriptive. The next useful experiment is to test whether an enforced output schema reduces invalid responses on a separate held-out sample while preserving correct choices.