Verified RAG Delivery Kit

A production-ready FastAPI + Next.js system for agencies and developers who build and sell Retrieval-Augmented Generation (RAG) chatbots to clients. It solves the problem every RAG freelancer hits after the demo: proving to a client that the bot actually works, doesn't hallucinate, and won't fall apart in production.

Ships as two distinct, fully-implemented codebases — not one codebase with cosmetic license tiers. Pick the one that matches how you work:

• Developer edition: a single-user, project-scoped backend. You manage your own isolated Projects, documents, chats, and test suites directly — no organizations, teams, or invitations. The leaner codebase, built for solo freelancers and small dev shops running their own client engagements under one account.
• Agency edition: a full multi-tenant backend with a Super Admin → Organization Admin → Member hierarchy. A Super Admin provisions client Organizations; each Organization Admin invites team members, manages RAG projects, and runs evaluations; Members get scoped, read-mostly access. Built for agencies onboarding client teams or internal staff with role-based access inside one deployment.

Both editions ship the same core RAG engine and the same two Next.js frontend applications (a client chat portal and an admin panel — Agency's admin panel additionally exposes organization/tenant management). Documents are ingested asynchronously through a Celery + Redis pipeline into PGVector, queried through hybrid (semantic + keyword) search with Reciprocal Rank Fusion, and answered through a citation-grounded chat endpoint that streams over SSE and strips or flags any citation the LLM invents. On top of that sits a batch evaluation system — the part most RAG boilerplates skip — that runs test suites against golden answers, expected sources, and forbidden claims, then generates a branded PDF client acceptance report you can hand over as proof of delivery.

Core capabilities (both editions)

• Per-project RAG configuration: chunk size/overlap, embedding model, generation LLM, Top-K, and similarity threshold — enforced at query time so a project can never see another project's data
• Async document ingestion pipeline (Celery + Redis): PDF/DOCX parsing, chunking, embedding, and PGVector storage with an ivfflat index, with automatic status tracking (UPLOADED → PROCESSING → READY/FAILED) and reprocessing support
• Hybrid retrieval: PGVector cosine similarity search combined with PostgreSQL full-text keyword search via Reciprocal Rank Fusion
• Streaming chat over Server-Sent Events with a regex-based citation grounding parser that cross-references every [n] citation against retrieved chunks and flags or strips unsupported claims
• Batch evaluation engine: define test cases (query, golden answer, expected sources, forbidden claims), run them across the full pipeline, and grade groundedness with a secondary evaluator LLM, plus latency/cost tracking per test
• Branded PDF client acceptance reports summarizing ingested documents, pass rates, cost, latency, and the frozen configuration used
• Webhook/Slack escalation dispatchers that fire automatically on ungrounded or out-of-scope runtime answers
• Pluggable multi-provider LLM and embedding config (OpenAI, Gemini, Claude, or local sentence-transformers) with encrypted per-account API key storage
• Built-in subscription/plan system with Stripe Checkout, token quotas, and top-up packs, plus audit logging on sensitive actions

What's different in Agency

• Deep multi-tenancy: Organizations → Projects → Documents/Chats/Test Suites, with granular RBAC (Super Admin, Org Admin, Member) and email invitations
• Per-organization Stripe subscriptions, token quotas, and top-ups (vs. per-user in Developer)
• Superadmin console for provisioning and auditing every client organization on the platform
• White-label client handover and resale rights