Can small AI models diagnose machine faults?
I tested four small language models to see how well they answered industrial fault-diagnosis questions on local hardware. Two things mattered: whether an answer was correct, and whether the software could read it. The model with the highest accuracy still answered nearly half the questions incorrectly.
What was tested
The models were Nemotron-3 Nano 4B, Gemma 3 4B, Phi-4-mini, and Phi-4-mini-reasoning. These roughly four-billion-parameter models ran through Ollama on a workstation. The experiment did not use a device installed beside machinery or test a working maintenance assistant.
The questions came from IBM Research's FailureSensorIQ benchmark. I used 2,667 single-answer multiple-choice questions across ten asset classes, including motors, pumps, compressors, and gas turbines. They ask about relationships between sensors and failure modes; they are not live sensor traces or records of maintenance interventions.
Each response was checked for the correct option and for the required machine-readable format. An answer in the right format can still be wrong. An answer in the wrong format presents a separate integration problem because the application cannot use it. The study required unusable output to stay within a 5% threshold before considering a configuration for further work.
I ran the full question set sequentially to look for serving problems that a short test could miss. Where clean comparisons were available, I changed temperature—the setting controlling sampling randomness—while holding model weights fixed. Nemotron ran at temperature 0.6.
Accuracy and usable answers
| Model | Temperature | Accuracy | Unusable |
|---|---|---|---|
| Nemotron-3 Nano 4B | 0.6 | 51.8% | 0.0% |
| Gemma 3 4B | 0.3 | 42.1% | 0.0% |
| Gemma 3 4B | 1.0 | 41.9% | 0.0% |
| Phi-4-mini | 0.3 | 36.6% | 4.9% |
| Phi-4-mini | 0.8 | 30.7% | 6.7% |
Nemotron answered 51.8% correctly and produced usable output throughout its full-set run. For context, the benchmark reports approximately 27.5% for blind guessing and a 60.2% human-expert mean. Those are published reference figures, not a separate human evaluation conducted for this study.
Gemma also produced usable output throughout, but its accuracy was lower: 42.1% at temperature 0.3 and 41.9% at 1.0. The small difference offers little reason to treat either temperature as an important accuracy improvement in this test.
Phi-4-mini showed a larger change. At temperature 0.3, both accuracy and usable-output rate improved over the 0.8 setting, bringing unusable output to 4.9%, within the study's threshold. This was a result for that model and task, not evidence that lowering temperature always helps.
Why the reasoning model is missing from the table
Phi-4-mini-reasoning did not produce a clean full-set result. A separate 240-question diagnostic on one asset class, at temperature 0.3, produced 40% unusable output. It cannot be fairly ranked against models that completed the full benchmark.
Some questions that failed late in a long run answered cleanly after the model was reloaded. That suggests the serving process degraded over time. Pressure on its cached working state is one possible explanation, but the mechanism was not confirmed. The experiment also did not isolate Nemotron's architecture as the reason it completed its run successfully.
Conclusion
Nemotron had the highest accuracy among the configurations with clean full-set results, while both Nemotron and Gemma consistently returned readable answers. That makes Nemotron worth testing further on specific equipment categories and over longer serving runs. Its error rate is still too large to treat this benchmark result as evidence that it can independently guide maintenance decisions.
Gemma's reliable formatting is useful evidence about integration, not a reason to overlook its lower accuracy. Phi-4-mini-reasoning needs a clean full-set run before its accuracy can be compared fairly. Any follow-up assistant would also need grounding or human confirmation of its answers. This study, whose hypotheses were not pre-registered, establishes a benchmark comparison rather than an improvement in real maintenance outcomes.