Microsoft-Decision-1 on Industrial Questions: Consistent Answers, Persistent Errors
A new type of AI model
In October 2026, Microsoft introduced Microsoft-Decision-1, a model designed for making structured decisions rather than generating conversational responses.
Built by further training the Qwen3.5-9B language model, Decision-1 focuses on classification, routing, prioritization, and decision scoring.
Unlike a conventional generative language model that produces written explanations, Decision-1 evaluates a defined question and returns a structured result. It can select an option from a supplied list, evaluate a yes-or-no condition, or assign a numerical score. For multiple-choice decisions, it also returns probabilities for each option.
This approach has interesting implications for industrial software. Many industrial workflows involve bounded decisions: identifying a sensor associated with equipment failure, classifying maintenance requests, or determining which operational condition requires attention.
Rather than asking a general-purpose language model to explain its reasoning, an application could use a specialized decision model to select from known alternatives.
But how reliable are these decisions when applied to industrial engineering questions?
The experiment
I evaluated Microsoft-Decision-1 version 1 using IBM Research's FailureSensorIQ, a benchmark covering relationships between industrial sensors and equipment failure modes.
The study used 487 two-option questions across ten equipment classes, including pumps, compressors, electric motors, generators, and turbines.
The objective was to measure two things: how accurately Decision-1 selected the benchmark answer, and whether its selections remained consistent when the answer options were reversed.
Using Microsoft Foundry, I evaluated the questions in their original order, then reversed options A and B and evaluated the same questions five times. The questions and answer content remained unchanged.
This allowed me to examine both answer-order sensitivity and consistency across repeated evaluations.
What I found
1. High accuracy on industrial questions
Decision-1 matched 457 of 487 benchmark answers (93.84%) in the original evaluation.
With the options reversed, accuracy remained between 92.81% and 93.63% across five evaluations, averaging 93.22%.
The model also preserved its original answer choice, regardless of whether that answer appeared as A or B, on approximately 97% of questions in each reversed-order evaluation.
This indicates substantial answer-order stability, although the original and reversed evaluations were conducted at different times, preventing a controlled causal comparison.
2. Consistency did not guarantee correctness
The more interesting finding emerged from examining individual questions across the five repeated evaluations.
| Outcome | Questions |
|---|---|
| Correct in all five evaluations | 449 |
| Incorrect in all five evaluations | 30 |
| Changed between correct and incorrect | 8 |
Decision-1 selected exactly the same answer text across all five repetitions for 479 of the 487 questions (98.36%).
However, this included 30 questions where the model consistently disagreed with the benchmark answer.
The distinction is important. A model that repeatedly produces the same answer may appear reliable, but consistency alone does not establish correctness.
3. Performance varied across equipment types
Accuracy was not uniform across the benchmark. Decision-1 matched every answer for aero gas turbines and reciprocating internal combustion engines, while steam-turbine questions achieved only 69.23% accuracy.
These differences suggest that aggregate accuracy can conceal weaker performance in particular question groups. The equipment groups also contain different question mixes, so the results do not establish that equipment type itself caused the differences.
What does this mean for industrial AI?
Decision-1 demonstrated strong benchmark agreement and highly repeatable selections on this limited industrial question set.
However, the findings highlight an important distinction between a model's ability to make a consistent decision and its ability to make a correct engineering judgment.
This study also has boundaries. Most questions concerned irrelevant sensor–failure relationships, the benchmark labels were not independently re-adjudicated, and possible overlap with the model's training data is unknown. The results therefore cannot establish performance in live industrial environments.
For industrial decision-support applications, specialized decision models may offer a useful way to select among predefined alternatives. But their outputs still need validation against domain knowledge, particularly when an incorrect choice could have operational consequences.
Conclusion
Microsoft-Decision-1 performed well on the tested industrial benchmark, demonstrating high accuracy and strong consistency even when answer positions changed.
The important finding is that consistency and correctness are not the same thing. Some incorrect answers remained just as stable as correct ones.
The next step is to examine these persistent disagreements with industrial domain experts and evaluate the model on unseen questions supported by engineering references.
Decision models introduce an interesting architectural possibility for industrial AI: separating structured decision-making from open-ended language generation. Understanding where those decisions can be trusted—and where they require further validation—is the question worth pursuing.
Microsoft-Decision-1 · FailureSensorIQ dataset
This is independent, self-funded personal research. The views expressed are my own and do not represent my employer or any model or service provider.