Can consistent AI classifications be trusted?
If an AI model gives a detailed maintenance classification and then independently chooses the matching broad category, is the detailed answer more likely to be right? I tested that idea repeatedly because a useful reliability signal should survive more than one favorable run.
Testing agreement without treating it as truth
The study used 108 short maintenance records from the MaintIE benchmark, containing 329 eligible fine-grained labels. Each record was sent to a model in three isolated calls. One requested a detailed classification, another requested a broad parent category, and a third repeated the detailed task using different wording. No call saw another answer, and the gold label never entered the prompt.
The main comparison asked whether the detailed answer was more accurate when its parent matched the separately produced broad answer. Correctness still came only from the detailed answer's exact match with MaintIE's gold label; agreement never counted as correctness by itself.
An earlier discovery experiment found a positive relationship for Qwen 3.5 122B and Gemma 4 31B, while the Astra estimate was uncertain. The new experiment fixed five fresh runs per model before seeing their outcomes. That produced 15 runs and 4,860 logical calls. The discovery run did not count as one of the five replications.
What remained stable across runs
The rule fixed in advance required at least four of five estimable runs, at least four positive accuracy separations, and a crossed run-and-record bootstrap interval for the mean above zero. Qwen and Gemma each exceeded it: all five runs were estimable, every run was positive, and each mean interval stayed above zero. Astra was kept outside that panel decision, but its five comparison runs also had positive mean separation.
| Model | Overall accuracy | Accuracy if consistent | Accuracy if inconsistent | Separation |
|---|---|---|---|---|
| Astra | 71.4% | 74.7% | 59.5% | 0.152 [0.038, 0.263] |
| Qwen 3.5 122B | 43.4% | 64.0% | 33.7% | 0.303 [0.185, 0.416] |
| Gemma 4 31B | 54.1% | 65.0% | 44.1% | 0.209 [0.100, 0.320] |
The direction was repeatable, but the size differences are descriptive rather than a tested ranking between models. Qwen had the largest separation while also having the lowest overall accuracy and valid-output coverage. Astra had the highest overall accuracy and the smallest separation. These observed differences do not establish a common confidence scale that can be transferred unchanged between models.
Output validity also mattered. Astra produced no invalid calls. Gemma produced five invalid-label calls in each run, while Qwen produced between 11 and 23 invalid or truncated calls per run. The overall accuracy calculation counted invalid detailed answers as wrong instead of quietly removing them.
The consistent answers still included mistakes
Agreement enriched accuracy; it did not guarantee it. Even the consistent subsets were only 74.7% accurate for Astra, 64.0% for Qwen, and 65.0% for Gemma on average. A system that accepted every consistent answer would still accept many wrong classifications.
As a second comparison, the models answered the detailed question again with independent wording. For Qwen and Gemma, this repeated-question agreement produced a larger point-estimate accuracy separation in all ten runs that met the evidence rule. Astra's disagreement groups were too small under that rule, so its repeated-question results remain descriptive. In any case, a larger separation alone does not establish a better operational policy: coverage, false acceptance, call cost, and the consequence of an error would also matter.
Why the maintenance context matters
I also annotated the maintenance records from an Industrial Engineering perspective, preserving the reasoning behind each judgment. This was a single-expert layer: I was both the annotator and the study author, and there was no inter-rater reliability check. Under my explicit authorization, an AI assistant derived context sufficiency for four of the 15 usable records, operational consequence for one, and specificity for three from my recorded rationale; the first seven records were also converted from an earlier “operational ambiguity” schema. Record-level coding was kept separate from model outcomes, although I had previously seen aggregate results.
Only 15 expert-reviewed records could be matched exactly to the evaluation data, so this part of the study is exploratory. Across all three models, mean accuracy was lower for each of the three comparisons fixed before analysis: high versus non-high operational consequence, high versus medium specificity required, and records marked as missing machine identity versus those without that code. The full subgroup results also show that the relationship between consistency and correctness was not stable across these small groups.
Among the six records rated high consequence, 23.1–28.6% of Astra's cross-level-consistent predictions were wrong across runs, compared with 40.0–45.5% for Qwen and 54.5% for Gemma. These are small-sample model-performance figures conditioned on an expert reading of short text. They are not estimates of plant risk, safety severity, or real-world failure probability.
The engineering rationale explains the limitation more clearly than the numbers alone. A short phrase can omit the machine, location, operating condition, or failure mechanism. Separate prompts can repeat the same plausible assumption; agreement cannot recover information that was never written down.
Conclusion
On this MaintIE evaluation, cross-level consistency repeatedly identified a subset of classifications with higher observed accuracy. That makes it a potentially useful screening signal, not a certificate that an answer is correct.
The practical next step is a prospectively designed decision study on another industrial dataset, with a larger independently annotated expert sample. It should compare acceptance and abstention policies using coverage, false acceptance, operational consequence, and review cost—not accuracy separation alone.