Testing the limits of the ordering formula
If an ordering formula makes little use of better labels, testing a stronger language model may not help. I investigated that possibility by removing the model entirely. The experiment supplied correct direction labels from the simulation and tested whether the ordering system could turn them into better results.
Testing perfect labels rather than another model
The preceding system mapped five demand-direction labels to safety-stock multipliers and passed them to an Order-Up-To formula. Its model results were close to the formula-only baseline. This left two possible problems: inaccurate labels, or a calculation that gained little from better labels.
To separate them, I used the simulation's own ground-truth schedule to supply direction labels. This “oracle” knew the simulation's categories perfectly; it did not know the optimal order quantity. Hand-written causal rules provided another source of labels. No language model ran in this experiment.
The fictional three-tier chain followed the earlier environment: 36 months of synthetic seasonal demand, simulated pandemic, conflict, and port-strike events, and random variation in lead times and fill rates. Each condition ran twenty times.
Fourteen variants changed the multiplier ranges, the meaning of a neutral label, order damping, forecast inputs, or the source of labels. Exponential smoothing, demand passthrough, and the unmodified Order-Up-To policy supplied comparisons. I measured stockouts and chain OVAR, the average across tiers of order variance divided by demand variance.
Before continuing to another language-model phase, a variant had to improve OVAR over Order-Up-To by at least 0.10 or come within 0.30 of exponential smoothing. This requirement gave the experiment a practical purpose: establish whether there was enough improvement in the ordering design to justify testing models within it.
What the alternatives changed
| Condition | Chain OVAR | Mean stockouts |
|---|---|---|
| Exponential smoothing | 1.1931 | 89.5 |
| Order-Up-To | 1.7527 | 87.3 |
| Oracle labels, original lookup | 1.7759 | 87.0 |
| Best candidate: neutral forecast-only order | 1.7334 | 89.5 |
The best candidate removed the safety-stock addition during neutral periods, leaving a forecast-only order. Its improvement over Order-Up-To was about 0.019, and its gap to smoothing was about 0.540. Neither was enough to meet the requirement for continuing.
Perfect labels with the original lookup produced a higher OVAR point estimate than the unmodified formula. Increasing the multiplier range also raised OVAR in the tested variants. A deterministic calendar rule reached 1.7493, slightly below the formula-only result, but again the improvement was too small.
Smoothing had much lower OVAR but averaged 89.5 stockouts, compared with 87.3 for Order-Up-To. It would therefore be misleading to describe lower variance as an improvement on every operational measure.
Conclusion
None of the fourteen tested variants improved enough to justify the planned language-model phase. Even correct direction labels failed to improve the original label-to-buffer design. For this line of experiments, changing the classifier was not the next step supported by the evidence.
That finding is limited to these variants, one supply-chain topology, and one synthetic demand profile with stochastic supply conditions. The hypotheses were documented before execution but not externally pre-registered. The oracle supplied correct categories, not optimal actions, so the result cannot rule out every possible ordering design.
The next experiment changed what the model controlled: the forecast's smoothing coefficient rather than a safety-stock multiplier. That tested a different way of using the model instead of repeating the same classification approach.