Part 11 · Building AI products
RAG Chatbot Handover Checklist for Client Go-Live
Give clients a repeatable acceptance test, a deployment and recovery runbook, named owners, and a clear go-live decision.
BoilerShelf Team
November 16, 2026 · 4 min read

Short answer: A RAG chatbot handover needs a release decision the client can reproduce. Agree on scope, source precedence and acceptance criteria before the build. At go-live, deliver the versioned test set, evidence from the release candidate, the operating runbook and a named owner for each unresolved item. Record the decision and the date of the next quality review.
What should you agree with the client before the build?
Describe supported users, languages, document types, and how fresh answers must be. Record the authoritative source when documents conflict. Build the test set with the client. Start with distinct cases for answerable questions, unsupported questions, recently updated documents and permission boundaries. Reserve a client-reviewed holdout set for sign-off, and keep its source references and expected outcomes versioned. Treat 30 to 50 questions as an illustrative pilot size. Increase coverage for higher-risk features and rare cases.
Sources: LangSmith RAG evaluation tutorial · NIST AI Risk Management Framework
Acceptance matrix you can copy
The figures below are a synthetic example. Agree on denominators, scoring rules and pass thresholds for the actual project. Critical access-control failures block release even if the aggregate answer score passes.
| Check | Illustrative criterion | Evidence |
|---|---|---|
| Evidence retrieval | Supporting passage in the top five for at least 18 of 20 answerable questions. | Query ID, expected passage ID and returned rank. |
| Grounded answer and citation | At least 18 of 20 answerable responses pass the agreed rubric for correctness, support and source citation. | Per-question rubric result and reviewed citation. |
| Abstention | At least 9 of 10 unsupported questions return an appropriate abstention or handoff. | Expected behavior and observed result. |
| Permission isolation | Zero unauthorized documents or citations in the agreed synthetic access-control cases. Any failure blocks release. | Case ID and reviewer confirmation of the boundary. |
Include pass, fail and skipped counts, the denominator for each metric, and descriptions of failed cases in the report. Log the application build, model ID, prompt and retrieval settings, corpus snapshot, test set version and run date. The sample thresholds are acceptance criteria for a test set and do not establish future production accuracy.
Delivery checklist you can copy
| Deliverable | Owner and evidence |
|---|---|
| Acceptance package | Project lead delivers the report, held-out cases and rerun instructions. Client approver records the decision and accepted exceptions. |
| Deployment and recovery | Ops owner delivers environment inventory, deploy and rollback instructions, recovery targets, and the result of a restore rehearsal. |
| Source updates and access | Content owner runs a sample add, update and removal, then checks the version in answers. Security owner verifies roles, support access and trace visibility. |
| Accounts and costs | Client and agency record account and billing owners, quota alerts and usage assumptions. Transfer access through the provider or secret manager and rotate credentials when needed. |
| Repository and support | Project lead records the release commit, dependency lockfiles and applicable licence references. Record the support channel, response expectations and the escalation or rollback owner. |
Sources: OWASP RAG Security Cheat Sheet · Google SRE on-call guidance
How do you make the go-live decision?
Ask the named client approver to record Go, Hold, or Go with accepted exceptions. Attach the acceptance report, agreed thresholds, unresolved issues, severity, owner and due date. Hold the release if a blocker fails or required evidence is missing. Agree who can disable or roll back the chatbot, how incidents are reported, and when the team will reassess known limitations.
Decision record to copy: Release version / evaluation run / result / client approver and date / accepted exceptions with owners and due dates / rollback owner / first review date.
At the first agreed post-launch review, rerun the pinned baseline before adding anonymized real queries to a new test-set version. Review failures, document freshness, spend and support issues, then assign follow-up actions.
Related guides
- RAG evaluation metrics: how to build the test set and what to measure.
- Multi-tenant RAG: keep client data separate: the data-handling questions a client will ask.
Frequently asked questions
How many test questions does a RAG chatbot need?
Start with 30 to 50 real questions written with the client, including questions the documents do not answer. Add the questions real users ask after launch.
Can I promise that the chatbot will never be wrong?
No. Promise measured thresholds on an agreed test set, state the known limits, and agree what happens when a wrong answer is reported.
Who should own the test set?
The client helps write it, and both sides keep a copy with instructions to rerun it. It is the only way to know whether quality changed after a document or model update.
Where to go next
If you deliver AI applications to several clients, the For agencies page explains how BoilerShelf agency editions are licensed for client work.
The idea in short
The scope, sources, test set and thresholds are agreed before the build. At the end the test set is run, an acceptance report shows every result including failures, the client receives the system, the accounts and the documentation, and signs acceptance. A review date reruns the test set with real questions added.
Patterns covered: Acceptance criteria · Test set · Acceptance report · Sign-off
- RAG
- Agencies
- Project delivery
- Evaluation
- Checklist

