AI Chatbot Pilot: How to Run a 30-Day POC That Actually Proves Something

Most companies do not fail at buying an assistant. They fail at proving one. An AI chatbot pilot gets kicked off with enthusiasm, runs for six weeks, produces a deck full of screenshots, and ends with a sentence nobody can act on: “it was quite good, mostly”. Then procurement asks what the business case is, and the room goes quiet.

The fix for a failing AI chatbot pilot is not a better model. It is deciding, in writing, what the pilot has to prove before anyone connects a single document. Here is a four-week structure that ends in a yes or a no rather than a shrug.

Why most AI chatbot pilots prove nothing

Three patterns cause almost every inconclusive pilot:

• Scope that grows to fit the enthusiasm. It starts as “HR questions in Slack” and by week three it is also handling IT tickets, customer refunds and the sales team’s pricing questions. Nothing gets good enough to judge.
• No baseline. If you never measured how many L1 questions your team handled before the pilot, you cannot claim a reduction afterwards. This is the single most common omission.
• Vibes as the success criterion. “Does it feel accurate?” is not a criterion. Twelve people will give you twelve answers, and the loudest one will decide your quarter.

None of these are technical problems, which is why swapping vendors rarely fixes them.

Before day one: agree what the AI chatbot pilot must prove

Write one page. Get it signed by whoever controls the budget. It contains five things:

1. One audience. Employees, or customers. Not both. Their questions, tolerance for error, and content sources are completely different.
2. One question set. Pull the last 200 real questions from your helpdesk, shared inbox or support queue. Not invented ones. This is your test set and it should exist before the vendor call.
3. A measured baseline. Current volume, current median time to answer, current cost per handled question. Rough is fine. Absent is not.
4. Numeric thresholds. For example: 70% of the 200 questions answered correctly with a citation, zero permission leaks, median response under three seconds.
5. A decision date and a decision maker. One named person, one date, two possible outcomes.

If you cannot fill in point three, spend a week measuring before you start. A pilot without a baseline is a demo with extra steps.

Week 1 — load real content, not the good content

The instinct is to feed the assistant your cleanest documentation. Resist it. A pilot on curated content proves that curated content works, which you already knew.

Load the sources your audience actually relies on, warts included: the help centre, the policy pages, the product docs, the one spreadsheet everyone quietly treats as the source of truth. Keep the volume tight — three to five sources beats thirty — but keep them real.

Two rules for week one:

• Exclude anything archived or superseded. Stale content is the leading cause of “the AI is wrong” complaints, and it is a content problem masquerading as a model problem.
• Wire identity in immediately. If the production system will be permission-aware, the pilot must be too. Retrofitting access control after a successful pilot is how pilots die in security review. See enterprise chatbot integrations: SSO, SAP, Salesforce and internal APIs.

Week 2 — tune retrieval, not the prompt

When answers are wrong, teams reach for the system prompt. It is the visible dial, so it gets turned. It is almost never the problem.

In a grounded assistant, a bad answer usually means the right passage never reached the model. That is a retrieval failure, and it has different fixes: chunking that respects document structure, hybrid keyword-and-vector search so exact terms like error codes and SKUs still match, reranking, and a recency weight so last month’s page beats the 2023 duplicate. Our explainer on retrieval-augmented generation covers why this is where the leverage sits.

Set the confidence threshold in this week too. An assistant that says “I could not find this, here is a human” for 15% of questions is far more valuable than one that answers everything with 80% accuracy. Confident wrong answers are the failure mode that ends internal rollouts.

Week 3 — put it where the questions already are

Adoption is not an answer-quality problem. A brilliant assistant on a page nobody visits will show you a beautiful accuracy score and no usage at all.

Deploy to the surface your audience is already in: Slack or Teams for employees, the website widget or WhatsApp for customers, inside the docs for developers. One retrieval pipeline behind several surfaces, not separate bots. The trade-offs are laid out in choosing AI chatbot channels and, for customer-facing pilots, the website chatbot guide.

Tell users it is a pilot. Give them one feedback control — thumbs down is enough. Do not build a survey.

Week 4 — measure the four numbers that matter

Run your 200-question test set, then pull the live data. Four numbers decide whether the AI chatbot pilot succeeded.

Number	How to calculate it	Reasonable threshold
Grounded accuracy	Correct answers with a valid citation, divided by questions answered, scored by a human against the 200-question set	70% or better in week 4
Containment	Conversations resolved without a human, divided by total conversations	Compare against your baseline, not against a vendor benchmark
Honest refusal rate	Escalations where the content genuinely did not exist, divided by all escalations	High is good — it means the confidence threshold is working
Coverage gaps	Count of distinct unanswered questions, clustered by topic	A ranked backlog, not a failure list

That last row is worth the pilot on its own. Even if you decide not to buy, you leave with a prioritised list of the documentation your organisation is missing, ranked by how often people ask for it.

For what to instrument beyond week four, see monitoring internal AI assistants.

Five ways an AI chatbot pilot gets quietly sabotaged

• The vendor loads the content. If you never learn how ingestion behaves on your messy sources, you have not tested the thing you are buying.
• Only enthusiasts use it. Volunteers ask easy questions. Include the sceptics; their questions are the real test set.
• Success gets redefined mid-flight. This is what the signed one-pager prevents.
• The pilot runs on a plan you would never buy. Check that the pricing model scales the way you would actually deploy it — the four models are broken down in our guide to enterprise chatbot pricing.
• Nobody owns the content afterwards. A pilot creates a documentation backlog. Assign the owner in week one, not in month four.

Who should be in the room, and what it should cost

An AI chatbot pilot needs four people and no steering committee. A content owner who can fix documentation the moment a gap appears. An IT or security contact who can approve the SSO connection in week one rather than week five. A frontline lead from the audience you chose, because they know which questions are actually hard. And the budget holder, who signs the one-pager and shows up on the decision date.

On cost, the useful question is not what the pilot costs but whether the pilot runs on the pricing model you would live with in production. A pilot on a flat trial plan tells you nothing about what 40,000 conversations a month will cost, and per-resolution pricing behaves very differently from per-seat once volume moves. Model the production number during the pilot, not after it — the four common structures and their hidden costs are broken down in enterprise chatbot pricing, and the platform categories worth shortlisting are in best enterprise AI chatbot platforms.

One more thing to settle early: where the data lives and whether it can be used for model training. If your answer to either is “we should check”, check in week one. Security review is the most common place a successful pilot goes to die, and it is entirely avoidable.

Frequently asked questions

How long should an AI chatbot pilot run?

Four weeks of live use is enough to reach a decision if the scope is one audience and one content set. Longer pilots do not produce better data; they produce more meetings.

How many questions do we need for the test set?

Around 200 real, previously-asked questions. Fewer than 100 and a handful of edge cases will swing your accuracy score by ten points.

Should we run two vendors side by side?

Only if you can give both the same content, the same test set and the same channel. Otherwise you are comparing implementations, not products. Shortlist first using the platform categories, then bake off two at most.

What is a realistic accuracy number?

On real, uncurated content, 70–85% grounded accuracy in week four is a good pilot. Anyone promising 95% out of the box is either demoing on curated content or counting refusals as successes.

What if the pilot fails?

Then it worked. A four-week no is dramatically cheaper than an eighteen-month rollout, and the coverage-gap report is a genuine asset either way.

Next step

An AI chatbot pilot is cheap insurance against an expensive rollout. Write the one-pager first. Pull the 200 questions. Then pick two or three real sources and give an assistant ten working days on them. If you want to start this week, request IntelloWork access — a workspace usually takes about a day, and you can point it at a live wiki or help centre on day one. If your content sits in Atlassian, start with the Confluence setup guide; if it is developer-facing, see AI chatbots for API documentation.