Measuring Software ROI With Control Lines: A Practical Guide

Plants are not laboratories: mix changes, seasons turn, three initiatives run at once. Here is how to attribute improvement honestly anyway, using an instrument every multi-line plant already owns.

Quick answer: The honest way to attribute improvement to a software system is a control comparison: roll out line by line, and measure covered lines against uncovered ones over the same period, per unit produced, under one definition. If covered lines gain three OEE points while uncovered lines gain one in the same season with the same order book, two points is a defensible claim.

The rollout sequence most plants treat as a compromise is actually their strongest measurement instrument, and it is free.

Why before-and-after lies

The naive method compares this quarter to last quarter and credits the software with the difference. But the difference also contains the new maintenance planner, the discontinued difficult SKU, the seasonal volume swing and the gearbox that finally got rebuilt. Before-and-after cannot separate the software's effect from everything else that changed, which is why both enthusiasts and skeptics can read the same delta and both be wrong. Attribution needs a comparison that experienced the same everything-else.

The control-line method, step by step

1. Freeze the baseline on all lines, covered and uncovered alike, with the same instrument and definition, over four to eight representative weeks. 2. Roll out in waves, not everywhere at once: the not-yet-covered lines are your control group.

Pick waves so covered and uncovered sets are comparable: same archetype, similar products, shared demand conditions. 3. Normalize per unit or per operating hour, at stated mix, so volume swings cannot masquerade as improvement. 4. Compare deltas, not levels: the claim is (covered improvement) minus (uncovered improvement) over the same window. 5.

Convert to currency with the plant's own margins, label cost-avoided separately from revenue-gained, and review the resulting ledger with finance quarterly. A saving finance has not accepted is a slide.

The honest complications

Spillover: uncovered lines sometimes improve because practices leak from covered ones (crews rotate, fixes propagate). This makes the method conservative: it understates the effect, which is the right direction to be wrong in. Small numbers: a two-line plant has a weak control; extend the baseline window and lean harder on per-intervention expected-versus-realised tracking instead.

And selection: if wave one deliberately took the worst lines, regression to the mean flatters the result; say so in the methodology and temper the claim. None of these break the method; they define how confidently to state its output. A fourth complication is human, not statistical: the crews on uncovered lines will ask why line 3 got the good stuff first.

Answer publicly and with the real reason (highest measured value, not favoritism), and give every line its date; control-line cooperation survives a sequence that has a public logic and dies under one that looks like politics.

What this buys you

Internally: improvement budgets aimed by evidence, and renewal or expansion decisions made from a finance-signed ledger rather than a satisfaction survey. Externally, if you are ever the one selling: a value claim that survives a diligence analyst, which in our experience is the rarest artifact in manufacturing software.

The whole apparatus costs almost nothing beyond discipline, because the rollout was going to be line-by-line anyway. The only change is refusing to waste the experiment it creates.

Frequently asked questions

What is a control line?

A comparable line not yet covered by the intervention, measured over the same period under the same definition. Its improvement represents everything-else; the covered lines' improvement beyond it is the attributable effect.

How do we normalize for product mix changes?

Express results per unit at stated mix, or restrict comparisons to periods and lines with similar mix. Where mix shifted materially, say so and bound the claim rather than adjusting it away silently.

Our plant has only two lines. Can we still attribute?

Weakly by control, better by forecasting: declare expected impact per intervention in advance and track expected versus realised. A track record of kept forecasts is attribution by another route.

Is this the same as M&V in energy projects?

Same spirit, lighter machinery. Energy M&V formalized baselines, adjustments and verification because money changes hands on the result; production software deserves the same discipline for the same reason.

Fabrico's rollouts are structured as covered-vs-control by default, with the value ledger reviewed quarterly. The full argument is Part 2 of our Proof, Not Promises series.