Finding out which part broke
Once a machine has broken down, its recent readings and error records may help identify the failed part. I tested whether language models could make that identification, and whether they offered anything over a trained classifier or a simple rule based on the latest error.
What the models were asked to identify
Every model was told that a breakdown had occurred and asked to choose which of four components had failed. This task is component attribution. It does not predict a future breakdown or establish the physical cause of a failure.
I used Microsoft's Azure Predictive Maintenance sample, which simulates 100 machines over 2015. It contains hourly voltage, rotation, pressure, and vibration readings, along with error codes, component replacements, and failure records. Selecting failures affecting a single component produced 677 events across 98 machines. The earlier 464 events were used for training, and the later 213 for evaluation. Machines could appear in both periods, so this tested later failures in the same fleet rather than transfer to unseen machines.
The models were gpt-5.4, gpt-oss:120b, and nemotron-3-super:120b. Nemotron was added to the main comparison after its results were visible. Its inclusion was therefore exploratory, and the original two-model conclusions do not depend on it.
For each event, the input was a text summary of evidence strictly before the failure: machine characteristics, three days of telemetry, a week of error records, and time since each component's replacement. One condition supplied only this evidence. The other added a fixed summary of failure rates and error frequencies from the training period.
Comparing models with simpler methods
Five baselines provided reference points: guessing according to failure frequencies, always guessing the most common component, choosing the oldest component, mapping the most recent error to a component, and a trained logistic-regression classifier. Predictions were scored against the simulator's labels, without a language-model judge.
The principal measure was macro-F1, which gives each component equal weight; accuracy was also recorded. Confidence intervals were calculated by resampling machines, keeping failures from the same machine together.
| Method | Evidence only | With history |
|---|---|---|
| gpt-5.4 | 0.440 [0.369, 0.508] | 0.908 [0.863, 0.947] |
| gpt-oss:120b | 0.366 [0.300, 0.427] | 0.883 [0.827, 0.929] |
| nemotron-3-super:120b | 0.355 [0.297, 0.414] | 0.855 [0.800, 0.902] |
The trained classifier scored 0.995 [0.983, 1.000]. The recent-error rule scored 0.923 [0.881, 0.959], and the oldest-component rule scored 0.467 [0.395, 0.534]. Adding training-period statistics substantially improved all three language models, but none had a higher observed score than the recent-error rule.
That last comparison is less decisive than the scores may suggest. The rule's interval overlaps those of the models with history, and no paired statistical test established that the rule was superior or equivalent to them. The classifier's interval, by contrast, did not overlap any language-model interval.
Logistic regression was the baseline selected before evaluation. An exploratory gradient-boosting alternative scored 1.000, but it was not the preselected comparator. The classifier also learned from labeled examples, while the language models received a summary, so these approaches did not have identical access to the training information.
Checking sensitivity to component names
A separate test renamed components and error codes for 51 events. The paired accuracy differences were zero for gpt-5.4, +0.059 for gpt-oss:120b, and −0.039 for Nemotron, where a positive difference means accuracy fell after renaming. This small test cannot establish that labels never matter or that the models had no prior exposure to relevant data.
The plan also called for a minimum-detectable-effect calculation, which would describe how large a difference the observed-data analysis could reliably detect. That calculation was not performed. This omission limits the probe and remains a disclosed departure from the method.
Conclusion
The trained classifier was the strongest method in the planned comparison for this simulated fleet. Historical statistics helped the language models considerably, but their results did not surpass the recent-error rule's observed score. The useful lesson is to test those simple alternatives before assuming the task needs a language model.
These results depend on the simulator's relationship between errors and component failures. They do not isolate reasoning ability or show that an assistant can safely diagnose real machinery. The hypotheses were stated before analysis but not formally pre-registered. Testing beyond this fleet and simulator remains necessary, with the same distinction between identifying a failed component and predicting a failure.