A pilot is a time-limited, small-scale launch of a solution using real data and real people, with success criteria written down in advance. Its results help decide whether to continue implementation, make changes, or stop.
A demo can show that the technology works in principle. A pilot checks whether it works for you: with your documents, scans, and handwritten notes in the margins; with your people, their habits, and their workarounds; in your process, where someone has to accept, check, and pass on the model's output.
A good pilot answers three questions. Is there enough data? What changes in people's work? How much will it cost after the pilot, when the solution has to be maintained?
A bot runs for a month. A month later, people say “it seems to work, people like it,” and the purchase decision is based on impressions. The pilot uses twenty clean documents, while the real work involves scans, handwritten notes, and spreadsheets in three formats. The pilot has no end date, so it is endlessly tweaked and quietly becomes a permanent system no one decided to adopt. Or the wrong thing is measured: the model is almost always right, but the employee's time per task has not changed.
In every case, money has been spent and no decision has been made. A failed pilot at least gives you an answer. The most expensive pilot is one that leaves you unsure what to do.
A large infrastructure company decided to buy a system one of its executives had seen at an industry expo: an operator could see the whole territory on screen, faults would be marked automatically from customer reports, and the system would predict which equipment was likely to fail based on previous years' records. It worked for the customer whose case was being demonstrated.
They ran a pilot in one department, and even that was expensive. It showed that the system depended on two things they did not have: detailed maps of remote areas and equipment data that no one had maintained for years. The forecasts never worked as intended. The pilot found the gaps, but no decision followed because no one had recorded in advance what counted as success. The system was modified and later rolled out further, without the feature it had been bought for. There was no AI involved. The mechanics were the same.
Which part to include: one workflow from the process map where the effect can be measured and a model error can be caught before it becomes costly. A pilot covering an entire department can itself cost as much as a small implementation. How the work is done now: time, error count, and volume, so there is something to compare against. Which criteria define a successful pilot, what their thresholds are, and which data will be used to check them. How long it runs. Who makes the final decision and which of three options they can choose: expand, revise, or close.
Also decide where a human in the loop remains during the pilot and what happens if the model makes a mistake. The failure points make this clear.
If the pilot succeeds, calculate the total cost of ownership: maintenance, updates, and people's time spent checking. A pilot often looks cheap because enthusiasts keep it running. There are not enough enthusiasts for an entire department.
Related terms: Process audit · Acceptance criteria · Total cost of ownership · Failure point
Planning a pilot? First define the question it should answer. Services →