TL;DR
The practical question was simple: can a language model small enough to run locally near industrial equipment help with fault diagnosis, without sending data to the cloud?
I tested four 4B-class models on IBM's public FailureSensorIQ benchmark and scored two things separately: accuracy and output reliability. The best clean full-set result came from Nemotron-3 Nano 4B: 51.8% accuracy with structured output on all 2,667 calls. That is well above blind guessing, but below the roughly 60.2% human-expert mean.
The right conclusion is not "autonomous diagnosis." It is "plausible local assistant, with engineering guardrails." The two most practical lessons were not the ranking itself: sampling temperature and serving reliability often decide whether a small model's output is usable at all.
Design and configuration
What was tested
FailureSensorIQ is an IBM Research multiple-choice QA benchmark for understanding relationships between industrial sensors and failure modes. I used its single-answer split: 2,667 questions across ten assets including gas turbines, transformers, motors, pumps, compressors, fans, generators, and reciprocating engines.
Each question asks for the best option from a labelled set. The benchmark reference points are useful: blind guessing is about 27.5%, always-answer-A is about 28.2%, the human-expert mean is about 60.2%, and the best expert is about 66.2%.
| Task | Single-answer industrial fault-diagnosis QA: choose the best sensor/failure-mode relationship from multiple options. |
| Dataset | IBM Research FailureSensorIQ, single-true multiple-choice split. |
| Scale | 2,667 questions across 10 industrial asset classes. |
| Scoring | Accuracy and integration reliability scored separately. Reliability means the output was well-formed and machine-readable. |
| Runtime | Local Ollama runs on a workstation. Edge claim is footprint-only, not a validated edge-hardware deployment. |
Results
Clean full-set runs
| Model | Type | Temp | n | Accuracy | Output reliability | Unusable |
|---|---|---|---|---|---|---|
| Nemotron-3 Nano 4B | Reasoning | 0.6 | 2,667 | 51.8% | 1.000 | 0.0% |
| Gemma 3 4B | Non-reasoning | 0.3 | 2,667 | 42.1% | 1.000 | 0.0% |
| Gemma 3 4B | Non-reasoning | 1.0 default | 2,667 | 41.9% | 1.000 | 0.0% |
| Phi-4-mini | Non-reasoning | 0.3 | 2,667 | 36.6% | 0.951 | 4.9% |
| Phi-4-mini | Non-reasoning | 0.8 default | 2,667 | 30.7% | 0.933 | 6.7% |
Phi-4-mini-reasoning is excluded from this leaderboard because it produced no clean full-set run. Its degraded 240-question diagnostic is treated below as a runtime failure case, not as a valid model score.
Key findings
What mattered in practice
Nemotron-3 Nano 4B was the best clean result, but not a decision-maker. Its 51.8% score is a real lift over guessing and came with perfectly clean output across 2,667 calls. It still sits below the expert mean, so the appropriate role is assistant or reasoning layer, not unsupervised maintenance authority.
Temperature was not a detail. Phi-4-mini moved from 30.7% at its 0.8 default to 36.6% at 0.3 on the same weights. Its unusable-output rate also moved from 6.7% to 4.9%, crossing the 5% reliability gate used in this baseline. This is a directional single-model contrast, not a universal law.
Temperature-insensitivity is useful too. Gemma 3 4B stayed essentially flat: 41.9% at default temperature and 42.1% at 0.3, with clean output on every call at both settings. A model that does not wobble when a setting changes is easier to commission.
Clean output and correct diagnosis are different axes. A model can be wired correctly but wrong, or knowledgeable but hard for software to read. In this kind of deployment, the parser and structured-output layer are part of the result.
The Phi-4-mini-reasoning failure was a serving-runtime issue. Its output degraded during a long run, but the same questions answered cleanly when re-sent to a freshly loaded model. That points to runtime state, not model weights. KV-cache pressure is the leading hypothesis, but the mechanism is not confirmed.
Commission per asset. One average hides large per-asset differences. A shop-floor assistant should be validated asset by asset, rather than accepted from a single headline score.
Runtime caution
The failure that should not be blamed on the model
Phi-4-mini-reasoning has no clean full-set result. Its default-temperature run broke, and the temperature-0.3 diagnostic covered only one asset with 240 questions and 40% unusable output. That result is useful as an operational warning, not as a fair comparison against Nemotron.
The evidence points to serving-state degradation: input was identical across the point where output began degrading, question difficulty did not predict failure, and questions that produced garbage late in the run answered cleanly when re-sent to a freshly loaded model.
The operational lesson is simple: monitor output reliability continuously and reload or reset the serving runtime on a schedule. A pilot can fail because the serving setup ages over a long run, even when the model itself is not the problem.
Practical implications
A practical local assistant, with guardrails
The baseline settles a design direction: Nemotron-3 Nano 4B is the primary small reasoning model to build around, with Gemma 3 4B at low temperature as a clean non-reasoning fallback. The bare model should not make the maintenance decision; it should surface likely causes for an engineer or a grounded system to confirm.
The engineering rules are less glamorous and more important: set the sampling temperature for the task, score reliability separately from accuracy, monitor serving health over long runs, reload the runtime before degradation becomes invisible, and commission the assistant per asset.
If the decision is a fixed threshold or a lookup, keep it in plain code. Use the model only for the judgement-shaped parts of diagnosis where language, symptoms, and sensor-failure relationships need to be connected.
Limitations
What this does and does not prove
This is a baseline characterisation, not a deployment. The top model is 2.8 GB and fits the footprint of an 8 GB edge box, but the runs were on a workstation. Treat this as footprint evidence, not validation on constrained edge hardware.
The temperature result is a single-model contrast. It is directionally useful for one-right-answer work, but it is not enough by itself to claim a general rule across models or tasks.
The architecture explanation for Nemotron's clean endurance is a hypothesis. Its Mamba-hybrid design plausibly carries less growing KV-cache state than a pure Transformer reasoning model, but this experiment compared models, not architecture in isolation.
Finally, the 51.8% top score is not good enough for autonomous maintenance action. The deployed use case needs grounding, reference data, retrieval, tools, or a human sign-off layer above the bare model.
Code, data, and sources
Dataset: IBM Research FailureSensorIQ on Hugging Face · IBM GitHub repository · arXiv:2506.03278.
Code and scored results: View code and results →
Disclaimer
Independent, self-funded personal research by Siddharth Srinivasan. Views are my own and do not represent my employer, any model or service provider, or any third party. The work uses publicly available data or synthetic scenarios; no proprietary employer or customer data is used.