TL;DR
On Microsoft's simulated predictive-maintenance dataset (100 machines, one year, 213 test breakdowns), the task was to name which of four components failed using only pre-failure evidence. A trained logistic-regression classifier scored 0.995 macro-F1 — a score that averages performance across all four components, so it cannot be inflated by always guessing the most common one. A one-line rule mapping the most recent error code to a component scored 0.923. The three language models scored 0.36–0.44 with no context, no better than guessing the most-worn part. Giving each model a one-page summary of the plant's history lifted them to 0.86–0.91, but none exceeded the one-line rule. A label-disguising probe found no contamination signal for two of the three models. For this shape of tabular sensor diagnosis, classical machine learning is the stronger approach.
The data and task
Setup
The Microsoft Azure Predictive Maintenance sample is a simulated dataset: 100 machines logged hourly for 2015, with four sensor channels (voltage, rotation, pressure, vibration), coded error events, component-replacement records, and breakdown records naming which of four components failed. Because it is simulated, every result here describes method behaviour on this simulator, not real-plant capability — the “ground truth” is the simulator’s own labels. After collapsing to single-component events I had 677 breakdowns over 98 machines, split by time into 464 for training and 213 for evaluation.
For each of the 213 evaluation breakdowns, the model is told a failure occurred and is shown the evidence strictly before it: the machine’s model and age, three days of telemetry summaries, a week of error codes, and each component’s time since last replacement. It must name the failed component. Scoring is exact match on the four options; there is no model acting as judge. I ran each language model in three conditions — zero-shot (evidence only), with-history (evidence plus a frozen summary of the training period’s statistics), and a disguised-names probe on 51 events that renames the components and error codes to test for reliance on memorised labels. Five non-model baselines set the floor and ceiling: prior-weighted random, majority class, oldest component, a train-fitted recent-error rule, and a trained logistic-regression classifier.
Results
Numeric results
| Method | Macro-F1 [95% CI] | Accuracy |
|---|---|---|
| Majority class | 0.140 [0.125, 0.155] | 0.390 |
| Oldest component | 0.467 [0.395, 0.534] | 0.474 |
| Recent-error rule | 0.923 [0.881, 0.959] | 0.930 |
| Trained classifier (logreg) | 0.995 [0.983, 1.000] | 0.995 |
| gpt-5.4 · zero-shot | 0.440 [0.369, 0.508] | 0.498 |
| gpt-5.4 · with-history | 0.908 [0.863, 0.947] | 0.920 |
| gpt-oss:120b · zero-shot | 0.366 [0.300, 0.427] | 0.408 |
| gpt-oss:120b · with-history | 0.883 [0.827, 0.929] | 0.897 |
| nemotron-3-super:120b · zero-shot | 0.355 [0.297, 0.414] | 0.380 |
| nemotron-3-super:120b · with-history | 0.855 [0.800, 0.902] | 0.859 |
Macro-F1 with 95% confidence intervals from a machine-cluster bootstrap (10,000 replicates, clustered by machine, 213 events). The disguised-names probe (51 events) is omitted from this table; it showed no contamination signal for gpt-5.4 and nemotron-3-super and a small effect for gpt-oss:120b.
What I found
Findings
- Zero-shot, the models did not outperform the deterministic baselines. All three beat the majority baseline but did not reliably exceed the oldest-component heuristic, and trailed the recent-error rule by a wide margin. My hypothesis that a zero-shot model would clear the baselines was not supported.
- The trained classifier outperformed every model, even with history. At 0.995 it sits above every language-model condition with non-overlapping confidence intervals (0.995 vs 0.908 / 0.883 / 0.855).
- Historical context accounted for the improvement. Supplying the training-period statistics added 0.47 to 0.52 macro-F1 across the models. This is consistent with the zero-shot gap being missing priors rather than missing reasoning — though this experiment does not isolate reasoning, so that reading is a hypothesis.
- No configuration exceeded the one-line rule. The best result, gpt-5.4 with history at 0.908, falls inside the recent-error rule’s confidence interval. I ran no equivalence test, so the precise statement is that the models did not exceed it.
- The result was not specific to one model. Two open-weight 120B models and a frontier model all landed in the same 0.86–0.91 band with history, and all trailed the baselines cold.
Why
Reading the result
This simulator builds a near-deterministic link between recent error codes and the failing component, which is why a logistic regression recovers it almost perfectly and a one-line rule captures most of it. A model reading the raw evidence cold has no access to that plant-specific mapping and performs near chance. The one-page statistics block closes most of the gap because it hands the model exactly the priors the classical methods learned — but it does not make the model better than the rule those priors imply. The disguised-names probe matters because this dataset appears in many public tutorials; renaming everything left gpt-5.4 unchanged and nemotron slightly better, with only GPT OSS 120B showing a small drop, so the with-history performance is largely genuine evidence-reading rather than recall. The practical takeaway: for tabular sensor diagnosis of this shape, a cheap classical model is the right tool, and a language model should be measured against the simple rule the data implies rather than assumed to beat it.
Code and data
Full design, frozen protocol, code, and scored results are on GitHub. View on GitHub →
Methodology note: the dataset is Microsoft-simulated — the “ground truth” is the simulator’s labels, so these results describe method behaviour on this simulator, not real-plant capability, and do not generalise beyond this one simulated, same-fleet task. The task is attribution conditional on a known failure, not failure detection or causal root-cause analysis. The disguised-names probe (n=51) bounds the contamination check to gross effects only. nemotron-3-super was added as a third model after its results were seen, a documented post-hoc choice that reinforces rather than changes the conclusions. Hypotheses were stated before analysis but not formally pre-registered.
This is my independent, self-funded personal research. Microsoft is not affiliated with or endorsing this work; I use the dataset under its MIT licence.