Research
Experimental research on LLM agent performance in industrial engineering decisions.
Published
PREDICTIVE MAINTENANCE · COMPONENT ATTRIBUTION · EXPERIMENT WRITEUP
On a Microsoft-simulated maintenance dataset, I tested whether a language model can name which of four components failed from pre-failure evidence. A trained classifier scored 0.995 macro-F1 and a one-line recent-error rule scored 0.923; zero-shot LLMs trailed at 0.36–0.44. Giving the models the plant’s training-period history lifted them to 0.86–0.91, but none exceeded the one-line rule. For this tabular sensor task, classical machine learning was the stronger approach.
MAINTENANCE & RELIABILITY · FAULT DIAGNOSIS · FAILURESENSORIQ
The question: can a model small enough to run locally — on a box beside the machines, no cloud — do real fault diagnosis, and what decides whether its output is usable?
Models tested: four 4B-class local models — Nemotron-3 Nano, Gemma 3 4B, and Phi-4-mini (standard and reasoning) — on IBM’s 2,667-question FailureSensorIQ benchmark.
The answer: a capable assistant, not a decision-maker. The best clean full-set run was Nemotron-3 Nano 4B at 51.8%, with structured output on all 2,667 calls — above the guessing floor, below the 60.2% human-expert mean. Usability hinged less on the model than on its setup: one temperature change moved Phi-4-mini by about six points, and a separate run was excluded after the serving runtime degraded mid-run.
SUPPLY CHAIN · ADAPTIVE SMOOTHING · EXPERIMENT WRITEUP
For the first time in my replenishment experiments, every AI condition produced OVAR below 1.0 — the AI was damping variance, not amplifying it. The key architectural change: the AI now controls the EMA smoothing parameter α inside the formula rather than a safety stock buffer on top of it. Best result: GPT OSS 120B blind, OVAR 0.535, marginally beating the fixed α=0.3 baseline at 0.545.
SUPPLY CHAIN · ORACLE ABLATION · EXPERIMENT WRITEUP
Perfect ground-truth labels, fed directly to the architecture with no LLM at all, produced worse OVAR than the Order-Up-To formula running alone. Fourteen architectural variants tested. None passed the gate. The best variant remained 0.540 OVAR above exponential smoothing — the same best-case gap as V4, unchanged by label quality, multiplier range, or formula variant. V5 closes the intent-classification line; V6 changes the control lever.
SUPPLY CHAIN · INTENT CLASSIFICATION · EXPERIMENT WRITEUP
Confining gpt-4.1-mini to one of five plain labels, with a fixed lookup table doing the arithmetic, made the output completely reliable — intent compliance 1.000 with zero fallbacks. But every condition still amplified order-swing 6 to 7× above exponential smoothing (chain OVAR 3.27–3.84 vs 0.545), and the three information conditions barely separated: the Equaliser Effect. The intermediate V4a variant, run on Azure at n=20; superseded by V4b WorldEvents as the primary V4 result.
SUPPLY CHAIN · INTENT CLASSIFICATION · EXPERIMENT WRITEUP
Four AI models — from a lightweight fast model to a 120-billion-parameter reasoning behemoth — produced supply chain outcomes that differed by less than one within-cell standard deviation across three information conditions. Replacing the AI’s float output with five discrete text labels fixed the calibration problem from V3b. It simultaneously created a structural bottleneck that none of the four tested models or prompt variants escaped.
SUPPLY CHAIN · HYBRID ARCHITECTURE · EXPERIMENT WRITEUP
Three AI models controlled the safety stock multiplier in a hybrid architecture; a mathematical formula handled the order quantity. All four hypotheses failed across three information conditions and 20 replications per condition. Every AI condition produced higher order variance than doing nothing. Context made two out of three models worse. Memory caused the advanced reasoning model to collapse.
SUPPLY CHAIN · SOVEREIGN MODEL · EXPERIMENT WRITEUP
India’s sovereign model showed no measurable difference from GPT OSS on this task: OVAR 4.504 vs. 4.52. Neither model detected Indian seasonal demand patterns. Exponential smoothing outperformed both by approximately 8×. The GPT OSS result is an Agentic Bullwhip Effect Version 2 context reference, not a co-run comparison.
SUPPLY CHAIN · EXPERIMENT WRITEUP
Four LLM configurations evaluated against three heuristic baselines across 20 replications and 11,520 LLM calls. Every heuristic outperformed every LLM on both order variance and stockout count simultaneously. All seven hypotheses rejected.
SUPPLY CHAIN · EXPERIMENT WRITEUP
All four configurations amplified demand variability. The context × reasoning condition produced a fully inverted tier pattern (OEM as the noisiest tier, Component as the quietest), reversing the standard upstream cascade.