AI implementation
Knowledge-base chatbots: prepare, test and hand over
The model is only one part of the system. Start with the information it may use, the questions it should answer and the point where a person takes over.
A chatbot can sound confident while answering from an out-of-date policy. Giving it more documents does not automatically solve that problem. Someone still needs to decide which information is current, who may see it and what the system should do when the answer is missing.
A knowledge-base chatbot uses a collection of business information to support its answers. In a retrieval-based setup, the system searches that collection before generating a response. This guide gives a practical preparation and testing method, with a clearly labelled fictional example rather than a claim about a deployed client system.
Give the first version one useful job
Choose a bounded customer task. For a home-services business, that might be explaining which areas the team covers and helping a visitor request a booking. It need not quote for complex work, promise an appointment or make changes to an account.
Write two lists: questions the chatbot may answer and actions it may not take. Ask the people who currently handle enquiries which questions repeat and which require judgement. The first version should be useful within those limits, rather than pretending to replace the entire team.
Our business chatbot overview discusses use cases and handover. Here the narrower question is how to prepare the knowledge and prove that the proposed answers are acceptable.
Build a small source register
For each document, record its owner, audience, update date and authoritative location. Begin with a small approved set: current service descriptions, booking policy, opening hours and contact instructions. Remove superseded copies from the live retrieval source rather than hoping the model will always choose the newest paragraph.
Keep public information apart from private customer records and staff-only procedures. Access restrictions must be enforced by the retrieval/application layer, not merely requested in a friendly system prompt. Microsoft's RAG preparation guidance covers the data and security constraints to consider.
The source register is also a maintenance plan. When opening hours change, who updates the source, when is the search index refreshed and who checks a test question afterwards? An unowned document becomes an unowned customer answer.
Download the chatbot preparation and test worksheet (TXT). Use it to name the first customer task, list approved sources and record expected answers before testing a system. The suggested test cases are planning prompts, not evidence that a chatbot has passed them. Keep completed copies and any private source details in your team's own workspace.
Test the boundary, not just the answer
- Approve
Name a current source and its owner.
- Retrieve
Find information the visitor may access.
- Check
Test relevance and source support.
- Hand over
Route uncertainty to the right person.
A worked example: an answer and a boundary
Fictional business and scripted acceptance example
Approved knowledge: “Example Heating visits North District on weekdays. Weekend visits need confirmation from the office. No emergency attendance is promised online.”
Visitor: “Can someone come on Saturday?”
Acceptable answer: “Weekend visits need confirmation from the office. I can show you the contact route, but I cannot confirm a Saturday appointment.”
Unacceptable answer: “Yes, Saturday appointments are available. You are booked.”
This is an expected-response specification, not a transcript from an AI model. In implementation, test whether the proposed system stays within the approved source and does not imply an action it has not performed.
A citation can help the visitor inspect a policy, but a link alone does not make the answer correct. The response must actually reflect the cited passage. If the available source contradicts another current document, return a controlled handover instead of silently choosing the more convenient answer.
Write the awkward test questions before launch
Keep a test set with the visitor question, allowed source, expected behaviour and result. Include ordinary questions, missing facts, ambiguous wording and requests to ignore the rules. Use synthetic customer details in development.
- Known answer: does it answer the question from the approved information?
- Missing answer: does it acknowledge the gap and give the correct contact route?
- Conflicting information: does it avoid presenting a guess as policy?
- Private information: can an anonymous visitor retrieve a restricted document?
- Action request: does it avoid claiming a booking, refund or account change it did not make?
- Source change: does the answer update after the knowledge is replaced and re-indexed?
Microsoft distinguishes retrieval quality, groundedness and answer relevance. Checking those separately helps explain a failure: the right document may not have been found, or the answer may have gone beyond the document it received.
Make the fallback useful for the customer
A handover should tell the person what the team needs and what happens next. Offer a normal form or email route that still works when the chat service is unavailable. Ask permission before forwarding conversation details and collect only what is necessary for that request.
For the fictional weekend-booking example, the fallback might ask for the preferred date and contact details, then clearly say the office must confirm availability. It should not repeatedly restart the chat or leave the customer wondering whether a request was saved.
If you add live actions later, treat them as a separate integration with permissions, validation and confirmation. A document about appointment policy is not an authenticated calendar connection.
Agree ownership after the demo
Before launch, name the people responsible for knowledge updates, failed-answer review, provider costs and outages. Set a spending control appropriate to the service, and monitor whether genuine enquiries reach the team. Do not use a conversation count as a substitute for useful outcomes.
Review failures with private information redacted. A growing collection of “the bot made it up” complaints needs changes to retrieval, source quality or allowed scope, rather than a more persuasive introduction. Microsoft's evaluation guidance also discusses adversarial and security tests; successful ordinary questions do not cover those risks.
The practical deliverable is more than a chat bubble: an approved source register, a repeatable test set, a working handover and a maintenance owner. Those are useful things to ask for in any chatbot proposal.
Sources and further reading
Technical references for this guide. The checklists and examples are our practical recommendations, adapted to the subject.

