AI SaaS Architecture: How to Structure a FastAPI and Next.js Product

An AI SaaS looks like any other SaaS from the outside: a login, a dashboard, a billing page. The differences sit underneath. Every request can cost real money, latency is measured in seconds rather than milliseconds, you depend on providers you do not control, and customers hand you sensitive data. This guide walks through the decisions that shape a FastAPI and Next.js product, in the order they usually bite.

1. Split the frontend and backend on purpose

Use Next.js for what it is good at: rendering, routing, SEO and the user interface. Keep the business logic, data access and every call to a language model in the backend. FastAPI suits that role because it is asynchronous, typed through Pydantic, and produces an OpenAPI description of your API for free. That description is your contract: the frontend, the documentation and your tests can all be generated or checked against it.

The rule that saves the most pain: the browser never talks to an LLM provider directly and never holds a provider key. All model traffic goes through your backend.

2. Authentication that survives production

Most early mistakes here come from convenience. Store short-lived access tokens, and keep the longer-lived refresh token in an HttpOnly cookie so scripts on the page cannot read it. Rotate the refresh token on every use and reject a token that has already been used, because a replayed token is the sign of a theft. Hash passwords with a slow algorithm, verify email addresses, and put roles behind every administrative route rather than hiding the buttons in the interface.

Decide early whether the frontend framework or the backend owns the session. Splitting responsibility between them is a common source of subtle bugs, such as a user who is logged in on one side and logged out on the other.

3. Choose your tenancy model before you have customers

There are three common shapes: single users with their own data, workspaces that a few people share, and organizations with roles, invitations and billing. Retrofitting tenancy into a product that assumed single users means touching every table and every query, so pick the shape you expect to need.

For shared databases, put a tenant identifier on every tenant-owned record and enforce it in one data-access layer instead of trusting each endpoint to remember. Then test it from the attacker's side: sign in as a member of tenant A and try to read tenant B's records by guessing identifiers. Every route should answer with a refusal.

4. Put an LLM gateway between your code and the providers

The gateway is a small internal service layer with one interface for chat and embedding calls, behind which sit the individual providers. It is where the AI-specific concerns live:

• Credentials. Store provider keys encrypted at rest, scoped to the tenant that owns them, and never return them to the browser.
• Model choice. Keep an allow-list of models per plan so a customer cannot select an expensive one by editing a request.
• Resilience. Apply timeouts, retries with backoff and a fallback provider, and translate each provider's errors into your own small set.
• Accounting. Record the model, input tokens and output tokens for every call. Use the usage numbers the provider returns, not an estimate based on text length.
• Limits. Enforce per-tenant rate limits and monthly quotas here, so no feature can bypass them.

Without a gateway, these concerns end up copied into every feature, and the first provider change becomes a refactor.

5. Move slow work to background jobs

A web request that waits twenty seconds for a document to be processed will time out behind a proxy and tie up a worker. Anything beyond a few seconds, such as document ingestion, batch runs and report generation, belongs in a queue with separate workers. Make tasks idempotent so a retry cannot do the work twice, store a status the frontend can poll, and decide up front what happens when a task fails halfway: retry, mark as failed with a reason, or alert a person.

6. Meter usage separately from billing

Keep two ledgers. Usage events record what happened: who, when, which model, how many tokens. Billing records what you charged. Deriving one from the other later is far easier than reconstructing usage from invoices. When a payment provider sends webhooks, verify the signature, process each event once, and grant entitlements only from the server side. Never trust a value the browser sends about what a customer has paid for.

7. Observability and cost control

Give every request an identifier and carry it through the logs, the queue and the provider call. Track latency percentiles instead of averages, because a slow tail is what users notice. Set a budget alert for provider spend, and cache only where the same input is guaranteed to deserve the same answer.

8. Deployment and secrets

Package the backend and the frontend as containers, run database migrations as an explicit step instead of at start-up, keep secrets in your platform's secret store rather than in the repository, and rehearse a restore from backup before you need one.

Build it or start from a codebase

None of these pieces is exotic, but together they take months to get right. If you would rather begin with them in place, compare the options by asking the same questions of any starter: how does it handle refresh tokens, how is tenant isolation enforced, where do provider keys live, what runs in the background, and what is not included? BoilerShelf's AI SaaS Core is one such starting point, and the SaaS boilerplates category lists the alternatives. Whichever you pick, run it and check its answers yourself.