BoilerShelf

Part 6 · Building AI products

Build a FastAPI LLM Gateway for Multiple Providers

Design one FastAPI interface for multiple LLM providers, with a shared deadline, capability-aware fallback and per-tenant usage accounting.

B

BoilerShelf Team

November 16, 2026 · 6 min read

A frosted teal gateway block with one incoming tile and three tiles fanning out.

Short answer: An LLM gateway gives FastAPI features one way to request a model by role. The gateway maps the role to an approved model, keeps the request within a deadline, and records the attempts and usage. Keep provider-specific prompts, response fields and error mappings inside adapters so the rest of the application can work with your own types.

How do you design one interface for several LLM providers?

Define the shape once, in your own types, so no provider's response object leaks into the rest of the code.

from dataclasses import dataclass
from typing import AsyncIterator, Protocol

@dataclass
class Usage:
    input_tokens: int
    output_tokens: int

@dataclass
class Completion:
    text: str
    model: str
    usage: Usage

class Provider(Protocol):
    async def complete(self, model: str, messages: list[dict], **opts) -> Completion: ...
    def stream(self, model: str, messages: list[dict], **opts) -> AsyncIterator[str]: ...

Each provider gets a small adapter class that turns your message list into its request and its response into a Completion. Adapters are where the differences live: how the system prompt is passed, what the usage fields are called, how a streamed chunk looks. Keep them thin and test each one against a recorded response.

As the gateway grows, change the streaming contract from AsyncIterator[str] to typed events for text deltas, tool-call deltas, usage and completion, and treat final usage as optional. An adapter should normalize request IDs, finish reasons and usage fields while keeping enough provider-specific information for billing reconciliation. For Gemini in new Python code, use the maintained google-genai SDK.

Sources: Google Gen AI SDKs · OpenAI Python SDK · Claude API errors

How should you route requests by role and capability?

Resolve a role such as fast, default or careful to an ordered list of eligible deployments. Filter by required features such as structured output, tool calls or vision, then enforce tenant and region restrictions. Validate structured responses against the same schema on each eligible route and evaluate fallback quality with the same fixtures. Load the routing policy from settings and record its version with each request.

An adapter library or a gateway such as LiteLLM can handle much of the provider translation and routing. Decide which layer owns retries, and keep application-specific decisions such as tenant limits and model eligibility in the application policy.

Sources: LiteLLM routing

How do you handle timeouts, retries and fallback?

Set one total deadline, with a shorter timeout for each provider attempt. Retry temporary connection failures, overload or eligible rate limits. A failed authentication check, invalid request or unsupported model feature needs a configuration or input fix. Let adapters parse Retry-After as seconds or an HTTP date and normalize it to a bounded delay input. If the wait cannot fit in the deadline, skip that retry. Review SDK defaults so retries do not multiply across layers. The OpenAI Python SDK, for example, retries certain failures twice by default and supports max_retries=0 when the gateway owns that policy.

The Python 3.11+ example assumes eligible_targets has already been filtered and invoke only calls the selected adapter. Keep usage persistence outside this loop. A timeout may occur after a provider accepted the request, so repeated attempts can incur separate charges. Use documented provider deduplication when available and keep attempt IDs for reconciliation.

import asyncio
import random
from collections.abc import Awaitable, Callable
from typing import TypeVar

T = TypeVar("T")


class TransientProviderError(Exception):
    """The adapter classified this failure as eligible for retry."""

    def __init__(self, retry_after_s: float = 0) -> None:
        super().__init__("Temporary provider failure")
        self.retry_after_s = retry_after_s


class GatewayUnavailable(Exception):
    pass


async def complete_with_fallback(
    eligible_targets: list[str],
    invoke: Callable[[str], Awaitable[T]],
    *,
    total_timeout_s: float = 12,
    attempt_timeout_s: float = 4,
) -> T:
    clock = asyncio.get_running_loop().time
    deadline = clock() + total_timeout_s
    last_error: Exception | None = None

    for target in eligible_targets:
        for attempt in range(2):  # Primary try plus one retry per target.
            remaining = deadline - clock()
            if remaining <= 0:
                raise GatewayUnavailable("Gateway deadline exceeded") from last_error

            try:
                async with asyncio.timeout(min(remaining, attempt_timeout_s)):
                    return await invoke(target)
            except (TransientProviderError, TimeoutError) as error:
                last_error = error

            if attempt == 1:
                break

            retry_after = getattr(last_error, "retry_after_s", 0)
            delay = max(retry_after, random.uniform(0.1, 0.3))
            if delay >= deadline - clock():
                break
            await asyncio.sleep(delay)

    raise GatewayUnavailable("Eligible providers unavailable") from last_error

For streams, retry or fall back only before any user-visible output or tool call has been delivered, unless the protocol explicitly supports resumption and deduplication. If the connection fails after a delta, report the generation as interrupted and keep the partial response. Treat cancelled work separately from provider failures. The streaming side is covered in stream LLM responses with FastAPI and SSE.

Sources: Retry-After HTTP semantics

How do you track LLM cost and enforce budgets per tenant?

Use a logical request ID and a unique attempt ID. Record the tenant, feature, actual provider and model, provider request ID when available, input and output usage by billable category, price snapshot, latency and status. Mark usage as reported, estimated or unavailable. A stopped stream may never deliver final usage, so preserve that uncertainty and reconcile it later. Record possible usage for failed attempts too. Protect the ledger from duplicate writes with a uniqueness constraint on the attempt ID.

Example with illustrative prices in USD per million tokens. If a provider reports 1,000 input tokens inclusive of 600 cached tokens, and 250 output tokens, charge 400 uncached input at $2, 600 cached input at $0.50, and 250 output at $8. The estimated cost is $0.0031. Apply this split only to APIs whose totals include the reported cached subset. Map cache creation, cache reads and any separately billed units according to each provider’s schema.

A pre-call query alone can overshoot a quota when several requests arrive together. Atomically reserve estimated capacity before dispatch, then reconcile against recorded attempt usage. Release unused capacity after settlement and expire abandoned reservations. Keep a recoverable event or ledger record if accounting storage fails after a provider succeeds. Avoid invoking the provider again just to retry the write.

Monitor retries, fallback rate, cancellation, time to first token, and spend estimates. Use provider request IDs or run IDs to investigate individual calls. Keep prompts, secrets and unneeded customer data out of default telemetry.

Sources: Claude cache usage fields · OpenTelemetry generative AI conventions

Frequently asked questions

Should I use a library or build my own gateway?

A library can give you the common interface and the routing. You still need your own usage records and limits, because they depend on your users, plans and prices. Many teams use a library for the provider layer and add a thin gateway module of their own on top.

Is it safe to fall back to a different model?

Only where a different model's output is acceptable. Chat usually tolerates it. A feature that needs an exact output format should fail loudly, unless you have tested the fallback model on it.

Where should model prices be stored?

In configuration or a database table with the date each price took effect, never in code. Store the computed cost on each usage row so that past numbers do not change when a price does.

Where to go next

For a complete FastAPI and Next.js starting point, see BoilerShelf AI SaaS Core.

The idea in short

A feature asks the gateway for a completion by role. The gateway maps the role to a provider and model, calls the provider through an adapter with a timeout, retries temporary failures, falls back to another model if allowed, and writes a usage row with tokens and cost before returning the result.

Patterns covered: LLM gateway · Provider adapters · Retry with backoff · Usage metering

  • LLM
  • FastAPI
  • Architecture
  • Python
  • Cost tracking

New guides and boilerplates, by email

Leave your email and we will tell you when a new guide or boilerplate is published.

All guides