The demo went well. The model read a messy customer email, filled in the order, and everyone in the room nodded. Three months later the pilot is still a pilot. Two people use it on good days. Nobody can say whether it saves time, and “should we roll this out?” keeps moving to the next meeting.
This is common. In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after the proof-of-concept stage by the end of 2025. It pointed to poor data quality, weak risk controls, rising costs and unclear business value. The model is rarely the main problem. The pilot was set up to impress, not to be measured and connected to real work.
This post covers why pilots stall, what has to change before an AI system is ready for daily use, and a checklist for making the move.
Demo, pilot, production
Three stages that often get mixed up:
- A demo shows that something is possible. Picked examples, one happy path, someone technical at the keyboard.
- A pilot tests whether it works on real work: real cases, one team, a person checking every output, a fixed end date.
- Production means people rely on it every day. If it breaks, work slows down, so someone has to notice and fix it.
Many “pilots” are really long demos. They never get real data, real users or a date when someone decides.
Why pilots stall
No baseline
If nobody measured how long the work took before, nobody can show that the pilot helps. The rollout then runs on enthusiasm, and enthusiasm runs out. Measure first; how to measure the ROI of an AI project shows how.
No owner
The pilot belongs to IT or an “innovation team”, while the people whose work it changes are too busy to care. Production needs an owner on the business side: the head of the team that does the work.
Toy data
The demo used ten clean examples. Real input is messier: scanned PDFs at an angle, emails in three languages, a customer who puts the order number in the subject line. A recruitment agency that tests CV parsing on twenty tidy CVs will be surprised by the twenty-first, a phone photo of a printed page.
No integration
In the pilot, people copy text into a chat window and paste the answer back into their system. That works for two weeks. In the first busy week it’s the first thing to go.
No way to tell whether a change helped
Someone tweaks the instructions to fix one case and quietly breaks three others. Without a fixed set of test cases, every change is a guess.
Unclear running cost
The pilot cost almost nothing. Nobody worked out what ten thousand cases a month would cost, or how much review time they need, so finance stops the rollout.
Users weren’t part of it
People who weren’t asked how they work see the tool as extra clicks, or as a threat to their job. They go back to the old way and usage falls quietly. Getting your team to actually use AI covers this side.
What changes in production
Pilot and production side by side
| Pilot | Production | |
|---|---|---|
| Users | One team, often volunteers | Everyone who does the work |
| Data | A sample, often cleaned up | Everything that arrives, as it arrives |
| Integration | Copy and paste, or a test system | Reads from and writes to live systems |
| Permissions | Broad, to move fast | Each user sees only what they may see |
| Quality checks | People eyeball the outputs | A test set runs on every change |
| When the AI fails | Someone happens to notice | A fallback route and an alert |
| Cost | Small, nobody tracks it | Measured per case, with a budget |
| Support | The builder, informally | A named owner and an agreement |
Integrations
In production the AI reads from your real systems and writes back to them: the mailbox, the ERP, the CRM, the document store. That means handling everything the demo skipped: duplicates, attachments the model can’t read, an ERP that is down for maintenance. Integration is often most of the build work. It’s also what turns a nice tool into a changed process.
Permissions and data
Pilots often run on an admin account. In production the AI should see only what the person using it may see, and only the data the process needs. If the AI can act (send, update, pay), keep write access narrow and require approval for anything hard to undo. Keeping AI agents safe goes deeper, and AI, GDPR and the EU AI Act covers the data protection side.
Evaluation
An evaluation set is a collection of real past cases with the correct result. For most business processes, 50 to 200 cases is a reasonable start, as long as they include the awkward ones. Every change to the instructions, the model or the reference data runs against the set before users see it.
The evaluation loop
- Real cases Past inputs with the correct result
- Run the new version Same cases every time
- Compare and score Rules check fields, a person reviews differences
- Ship or fix Only if it beats the last version
- Add new failures Mistakes from daily use join the set
The set also protects you when a provider updates or retires a model. You run it on the new model and know within the hour whether anything got worse. Why AI makes things up explains what to check for.
Instructions under version control
In a pilot, the instructions for the model (the prompt) often live in someone’s notes. In production they’re part of the system: stored with the code, changed through a review, tested against the evaluation set, and easy to roll back. The same goes for the choice of model and its settings. When something goes wrong on a Tuesday, you want to know exactly what changed on Monday.
Monitoring and fallbacks
Log every request: the input, the output, which model, how long it took, what it cost, and whether a person changed the result. Look at a few numbers every week: volume, share of outputs corrected, errors, cost per case.
Decide in advance what happens when the AI is unavailable or unsure. The case goes to a person through the old route. Nobody should be stuck because a model provider has an outage.
Cost control
Take the cost per case from the pilot logs and multiply it by your real volume. Set a monthly budget alert. Use smaller, cheaper models for simple steps such as sorting, and keep the larger ones for the hard steps. If you use an agent, cap how many steps it may take. What an AI project really costs covers the levers.
Support and ownership
Someone has to answer “why did it do that?” and fix it. Agree who: an internal developer, the vendor, or the team that built it. Write down who may change the instructions and how changes get tested. In our projects the client owns the code, repositories and accounts from launch, and we fix bugs for free for three months. Monthly support after that is optional. Whatever your setup, make it explicit before launch, not after the first incident.
A plan for the move
From pilot to daily use
- 1
Decision
Go or no-go with numbersTime per case, errors, usage and cost against the baseline.
- 2
Weeks 1–2
Build the evaluation set50–200 real cases, including the awkward ones.
- 3
Weeks 2–5
Connect to live systemsRead and write directly, with proper permissions.
- 4
Weeks 4–6
Add monitoring and a fallbackLogs, weekly numbers, a manual route when the AI can’t help.
- 5
Weeks 6–8
Roll out team by teamTrain each team on its own cases; people still review at first.
- 6
After launch
Hand over, review monthlyNamed owner, agreed support, review rules loosened as trust grows.
You don’t have to check every output forever. Once the numbers show the AI is reliable on a type of case, move to checking exceptions and a sample. Human in the loop describes the patterns, from reviewing everything to reviewing by exception.
An example: suggested replies in a support team
Take an online shop with a support team of eight that piloted AI-suggested replies. The example is illustrative. In the pilot, two agents copied customer emails into a chat assistant with a long prompt and pasted the drafts back into the helpdesk. They liked it and felt faster, but nobody had measured how long replies took before, and the other six agents never started.
Moving it to production meant:
- Integration: the draft appears inside the helpdesk ticket, already filled in with the order details from the shop system. No copying.
- Data: the model answers from the shop’s help articles and returns policy, not from memory, and each draft shows the article it used.
- Evaluation: 120 past tickets with the replies senior agents had sent became the test set. Every change to the instructions runs against it.
- Rules: refunds above a set amount, complaints and anything mentioning a lawyer never get a suggested reply. They go straight to the team lead.
- Monitoring: a weekly count of drafts sent with small edits, drafts rewritten and drafts thrown away.
- Training: a one-hour session per shift, using each agent’s own recent tickets.
None of this needed a better model. It’s plumbing, rules and habits, and that’s where most of the work goes. AI in customer support covers this use case in detail.
Checklist: ready for production?
Before you call it production
- A business owner Not only IT
- Baseline and pilot numbers Time, errors, usage, cost per case
- An evaluation set Run on every change
- Live integrations No copy and paste
- Least-privilege access Users see only what they may see
- Approval for risky actions Sending, paying, deleting
- Logs and weekly numbers Volume, corrections, errors, cost
- A fallback route Work goes on when the AI can’t help
- Trained users On their own real cases
- Support agreed Who fixes it, and how fast
What to do next
If you have a pilot stuck in limbo, set a decision date, pull together the numbers you have, and go through the checklist. The unticked boxes are your plan for the next four to eight weeks. If the numbers are missing, that’s the first box.
If you’re starting fresh, follow the 90-day plan and design the pilot with production in mind from the first day: real data, a baseline, an owner, and a decision date.
Turning pilots into tools people use every day is most of what our internal tools and workflow projects are about. We show a working demo every week, so problems show up early. If yours is stuck, tell us where.
