Gemini 4 Argon on IBM FailureSensorIQ: Testing at Provider Defaults
I tested Gemini 4 Argon alongside Gemini 3.8 Flash, Gemini 3.1 Pro preview and GPT-6.1 Sol on IBM’s FailureSensorIQ benchmark. Across the four models, I collected 702 answers. The main comparison uses the same 170 questions for every model.
What I tested
FailureSensorIQ contains engineering questions about machine faults and sensors. I used its original single-answer split, which has one released correct answer among two to five options per question. The shared subset covered electric motors, steam turbines, aero gas turbines and industrial gas turbines, with questions about sensors relevant to failure modes.
I kept IBM’s original question wording, reasoning prompt and answer parser. Generation controls were left unspecified so each provider applied its defaults. This differs from IBM’s reference runner, which sets temperature to zero.
The 170 shared questions give 680 matched model answers. The remaining 22 answers are retained in the supporting records. Scores below use only the shared questions, keeping the comparison consistent across models.
Performance on the same 170 questions
| Model | Correct answers | Accuracy | Multiple selections | Zero selections |
|---|---|---|---|---|
| Gemini 3.1 Pro preview | 93 / 170 | 54.71% | 8 | 2 |
| Gemini 3.8 Flash | 92 / 170 | 54.12% | 6 | 0 |
| GPT-6.1 Sol | 83 / 170 | 48.82% | 33 | 1 |
| Gemini 4 Argon | 73 / 170 | 42.94% | 63 | 1 |
Gemini 3.1 Pro preview had the highest count at 93 correct answers, followed by Flash at 92, Sol at 83 and Argon at 73. Pro’s one-question lead over Flash does not establish a clear general winner between them.
Why Argon’s multiple selections matter
Each question has one correct answer in IBM’s released key. A complete match scores one point; every other selection scores zero. Selecting the correct option together with an extra option also scores zero.
Argon selected multiple options on 63 of the 170 questions, compared with 33 for Sol, eight for Pro and six for Flash. Those answers therefore lost points under the strict scoring rule.
IBM’s native prompt asks for the correct “option(s)” and an answer list. I preserved that wording even though this split has one correct answer per question. The resulting score reflects both exact agreement with the key and how the model responds to that interface. Argon’s lower score does not, by itself, demonstrate weaker industrial knowledge.
Conclusion
At provider defaults, Gemini 4 Argon achieved 42.94% exact-answer accuracy on this subset, while Pro and Flash had similar observed scores near 54%. Argon’s frequent multiple selections are central to interpreting that difference.
This was a single pass on the first 170 shared questions, covering four equipment classes and sensor selection. The findings are exploratory and do not establish a general ranking of industrial reasoning ability. A separate study could test whether explicitly requesting one option changes the multiple-selection pattern.