Part 5 · Building AI products
How to Stream LLM Responses With FastAPI and SSE
Build a FastAPI SSE endpoint and a Next.js reader. See how to stop a generation, keep partial output and test streaming behind a proxy.
BoilerShelf Team
November 16, 2026 · 5 min read

Short answer: Stream LLM responses to Next.js by sending the prompt in a POST request and returning a text/event-stream response from FastAPI. Use the browser’s fetch API to read the stream, and wire the Stop button to an AbortController. The server must also close the upstream provider stream. Start with a deterministic mock so you can see all three pieces work before adding a model.
Why use Server-Sent Events instead of WebSockets for LLM streaming?
A chat answer flows one way, from server to browser. SSE is plain HTTP and needs no connection state on the server. It usually passes through load balancers, but a proxy may buffer it, which is covered below. WebSockets earn their extra work when the browser also needs to push data during the response, such as live audio.
How do you stream an LLM response from FastAPI?
FastAPI 0.135.0 added EventSourceResponse and ServerSentEvent in fastapi.sse, and the example below uses them. On an older FastAPI version, return a StreamingResponse with the text/event-stream media type instead, and write the event framing and the keepalive messages yourself. SSE carries one-way updates over HTTP, so check that every proxy in front of the route forwards them as they arrive.
Sources: FastAPI SSE documentation
Save the mock as app.py and run uvicorn app:app --reload with FastAPI 0.135.0 or newer and Uvicorn installed. Each event holds one text delta. A delta may contain part of a word, multiple characters, or a newline.
import asyncio
import logging
from collections.abc import AsyncIterator
from contextlib import aclosing
from fastapi import FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel, Field
app = FastAPI()
log = logging.getLogger("uvicorn.error")
class ChatIn(BaseModel):
message: str = Field(min_length=1)
async def mock_tokens(prompt: str) -> AsyncIterator[str]:
try:
for piece in ("Echo: ", prompt, "\n", "Done."):
await asyncio.sleep(0.35)
yield piece
finally:
log.info("Mock stream closed")
@app.post("/chat/stream", response_class=EventSourceResponse)
async def chat_stream(body: ChatIn) -> AsyncIterator[ServerSentEvent]:
try:
async with aclosing(mock_tokens(body.message)) as stream:
async for delta in stream:
yield ServerSentEvent(event="token", data={"token": delta})
except asyncio.CancelledError:
raise
except Exception:
log.exception("Generation failed")
yield ServerSentEvent(event="error", data={"message": "Generation failed"})
return
yield ServerSentEvent(event="done", data={})
Validate input and access before starting the stream. After the response has started, report generation failures with an error event. Define token, done and error as named events, send done after successful generation, and let a disconnected client’s cancellation propagate.
How do you read the stream in Next.js and add a Stop button?
Install eventsource-parser and connect /api/chat/stream to FastAPI’s /chat/stream through a same-origin proxy that streams the upstream response. Align the paths with your deployment. Create a fresh AbortController for each generation and attach its signal to fetch.
import { EventSourceParserStream } from "eventsource-parser/stream";
export async function streamChat(
message: string,
onToken: (token: string) => void,
signal: AbortSignal,
) {
const response = await fetch("/api/chat/stream", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ message }),
signal,
});
const type = response.headers.get("content-type")?.split(";")[0].trim();
if (!response.ok || !response.body || type !== "text/event-stream") {
throw new Error(`Could not start stream (${response.status})`);
}
const reader = response.body
.pipeThrough(new TextDecoderStream())
.pipeThrough(new EventSourceParserStream())
.getReader();
try {
while (true) {
const { done, value } = await reader.read();
if (done) throw new Error("Stream ended before the done event");
if (value.event === "done") return;
if (value.event === "error") throw new Error("Generation failed");
if (value.event === "token") {
const { token } = JSON.parse(value.data);
if (typeof token !== "string") throw new Error("Invalid token event");
onToken(token);
}
}
} finally {
await reader.cancel().catch(() => {});
reader.releaseLock();
}
}
An SSE record may cross multiple network chunks. The parser handles carriage returns, repeated data fields and heartbeat comments. Show an interrupted state if the connection ends before done. Keep the text received so far when the user stops. Keep the active AbortController in a ref, call abort() from Stop or component cleanup, and guard UI updates with the active run ID so an old stream cannot update a new message.
Sources: Event stream format · eventsource-parser · AbortController
How do you stop the provider call and fix proxy buffering?
Replace the mock with the provider SDK’s documented async context manager or close method. Keep ownership of the upstream stream inside the generation task and close it when that task exits. If a separate worker owns generation, route cancellation to that worker. Record a run ID and distinguish completed, stopped and failed messages. Verify closure on Stop, browser disconnect, and a stop while the model is quiet. Closing the stream does not establish whether the provider stopped work immediately or how it bills the request.
FastAPI’s native response sends idle keepalives and sets X-Accel-Buffering: no. At every proxy hop, verify buffering, compression, idle read timeouts and request duration limits. In Nginx, proxy_read_timeout measures silence between reads. A heartbeat can help with idle timeouts. A hard maximum duration requires changes at the hosting layer. Test the public route with curl -N and compare first-event timing with a direct FastAPI request.
Sources: Next.js self-hosting and streaming · Nginx proxy directives
Related guides
- One LLM gateway for several providers in FastAPI: where model calls, retries and cost tracking belong once the stream works.
- AI SaaS architecture on FastAPI and Next.js: how streaming fits with authentication, tenancy and background jobs.
Frequently asked questions
Can I use EventSource with a POST request?
No. The built-in EventSource only sends GET requests and cannot set headers. Use fetch and read response.body as a stream, as shown above.
Do I need WebSockets for a chat interface?
Not for a one-way answer stream. SSE is plain HTTP and passes through most load balancers. WebSockets are worth the extra work when the browser must also send data during the response, such as live audio.
Why does the model keep running after I click Stop?
Aborting the fetch closes the connection, but the request to the model provider only ends if your server closes it when the response generator exits. Wrap the provider call so that leaving the loop closes the upstream stream, then check the provider's usage for a stopped request.
Where to go next
If you would rather start from a complete application than assemble these parts, see the BoilerShelf Streaming Chatbot.
The idea in short
The browser sends the prompt with a POST request. FastAPI returns an event stream that carries one JSON message per token, then a done or error event. If the user clicks Stop, the browser aborts the request, the server notices the disconnect and closes the upstream model stream.
Patterns covered: Server-Sent Events · Async generators · Cancellation · Heartbeat
- FastAPI
- Sse
- Streaming
- Next.js
- LLM

