How to Design a Manufacturing Software Pilot That Proves Something

Most pilots are demonstrations wearing lab coats: friendly line, vague goals, no end date, decision by impression. Here is the structure that turns a pilot into an instrument, including the one sentence to agree before anything is installed.

Quick answer: A pilot that proves something has five properties agreed in writing before installation: a frozen baseline (same instrument, same definition, before intervention), a fixed duration, defined artifacts it must deliver, an attribution method (usually the uncovered lines as controls), and an exit question whose answer decides go or no-go. If any of the five is missing, you are running a demo, and demos are decided by charisma.

The exit question comes first

Before scope, before line selection, both sides agree one sentence: what question will this pilot answer, with measured data, by which date? A good exit question is decidable and commercial: what does this line lose per month in currency, and which three actions recover the most of it?

A bad exit question is a mood: do we like the system? The exit question disciplines everything downstream, including the vendor, because a pilot that cannot answer its question has failed by its own definition, and both sides know what failure looks like from day one.

The five design decisions

Line selection: pick a line that matters and hurts, not the friendliest one. A pilot on the easy line proves the system works on easy lines.

Baseline: measure before improving, with the pilot's own instrument, for long enough to cover normal variation; and say out loud that the honest number will start lower than the reported one, because honest measurement catches what logs miss.

Duration: fixed, short enough to keep urgency, long enough to see pattern; four to eight weeks fits most lines. Open-ended pilots are where decisions go to die.

Artifacts: name the deliverables (the loss Pareto, the changeover self-comparison, the gap in currency) so completion is checkable.

Attribution: the lines not in the pilot are your control group for free; compare covered against uncovered over the same period before claiming any effect.

What each side owes

The plant owes access (the line, its signals, a few engineer hours), honest inputs (margins for the currency conversion), and a decision-maker who will actually attend the findings session. The vendor owes speed (measuring within days, not a mini-implementation), the artifacts as specified, and the findings whether they flatter the product or not.

Paid pilots align both sides better than free ones: the fee filters unserious buyers, funds serious vendor effort, and, credited against a subscription, costs a committed buyer nothing.

And one thing both sides owe the floor: the crews hear about the pilot before the hardware arrives, from their own leadership, with two commitments made explicitly. First, the honest baseline will read lower than the reported number, because it catches what logs miss, and that drop will not be blamed on anyone; the number got true, not worse.

Second, the measurement ranks problems, never people, in writing. Pilots done to operators produce compliance theater and quietly sabotaged sensors; pilots done with operators recruit the only people who already know where the losses are.

Reading the result

Three outcomes, all useful. The question is answered and the numbers justify continuing: proceed, with the pilot's baseline as year one's reference. The question is answered and the numbers do not justify it: stop, richer by one honest measurement and one avoided contract.

Or the pilot could not answer its question: also informative, usually about data access or vendor capability, and better learned in six weeks than eighteen months. The only failed pilot is the one that ends in a feeling.

Frequently asked questions

How long should a manufacturing software pilot be?

Four to eight weeks on one line covers normal variation without losing urgency. Shorter risks mistaking a good fortnight for a trend; longer usually signals a missing exit question.

Should pilots be free?

Free pilots select for casual buyers and casual vendor effort. A fixed fee, credited on continuation, keeps both sides serious and costs a committed buyer nothing.

How do we attribute improvement to the software?

Use the uncovered lines as controls over the same period, normalize per unit produced, and compare against the frozen baseline under the same definition. Improvement claims without a comparison group are weather reports.

What if the pilot shows our real OEE is much lower than reported?

Expect it: honest capture always finds the stops the logs missed. The number did not get worse; it got true, and the true number is the only usable starting point.

This structure is how Fabrico's Loss Profile Diagnostic is built: fixed scope, fixed fee, defined artifacts, exit question agreed up front. The proof machinery behind it is our Proof, Not Promises series.