Answer box
A working local AI stream does not prove that your deployed response reaches the browser incrementally. Your application, compression middleware or reverse proxy may buffer it; a hosting function may also terminate before it finishes. Browser code that waits for the whole response can produce the same visible symptom. Start with a short timestamp stream in staging, then compare what the server writes with what the browser receives. Change only the layer the test implicates. (expressjs.com)
1. Trace both streaming connections, not just the AI request
For a website that calls an AI provider through its backend, map the request path:
Browser → CDN/load balancer → reverse proxy → website backend → AI provider
Browser ← CDN/load balancer ← reverse proxy ← website backend ← AI provider
Treat this as two separate streaming connections: provider to backend, and backend to browser. Receiving provider events successfully does not establish that your application forwards them without accumulating the complete response. Cloudflare’s Workers documentation illustrates this distinction: preserving a response stream requires streaming its body rather than reading the entire body first. (developers.cloudflare.com)
Check the format on each connection. Server-sent events, or SSE, use Content-Type: text/event-stream, with events separated by blank lines. Native EventSource understands that framing. A Fetch-based client can instead read the response body incrementally, but your application must interpret its chosen format. (developer.mozilla.org)
Do not assume provider events are plain text fragments. For example, OpenAI’s Responses streaming guide distinguishes text-delta events, completion events and errors. Forward the documented event format, or deliberately translate it into a format your frontend understands. (developers.openai.com)
Before investigating infrastructure, inspect the browser handler. Calling await response.text() or await response.json() waits for the complete body. Use the response stream when you want incremental rendering. (developer.mozilla.org)
2. Replace the model with a harmless timestamp stream
First establish whether ordinary incremental output survives deployment.
Use staging with the same runtime, middleware and proxy path as the affected feature. Protect the diagnostic route with your staging access controls. Do not include prompts, credentials or customer data. Save the existing deployment and configuration so you can revert.
The following original diagnostic handler is for a traditional Node.js HTTP response, not every serverless or framework adapter. Mount it on a staging-only GET route such as /stream-check. Platforms using Web Response objects need an equivalent ReadableStream implementation.
Node documents flushHeaders(), repeated response writes and the response close event used here. Flushing headers is not a command to flush every downstream proxy. (nodejs.org)
export function streamCheck(req, res) {
res.writeHead(200, {
"Content-Type": "text/event-stream; charset=utf-8",
"Cache-Control": "no-store",
"X-Accel-Buffering": "no"
});
res.flushHeaders();
let count = 0;
const timer = setInterval(() => {
if (res.destroyed || res.writableEnded) {
clearInterval(timer);
return;
}
count += 1;
res.write(
`data: ${JSON.stringify({
sequence: count,
emittedAt: new Date().toISOString()
})}\n\n`
);
if (count === 10) {
clearInterval(timer);
res.end("event: done\ndata: complete\n\n");
}
}, 1000);
res.on("close", () => clearInterval(timer));
}
This test is designed to emit ten timestamp messages approximately one second apart. That schedule is a diagnostic input, not a measured hosting result.
For a same-origin browser check, use:
const stream = new EventSource("/stream-check");
stream.onmessage = (event) => {
console.log("received", new Date().toISOString(), event.data);
};
stream.addEventListener("done", () => stream.close());
stream.onerror = () => {
console.warn("Stream interrupted");
stream.close();
};
Closing on the explicit done event keeps this finite diagnostic from reconnecting automatically. SSE normally reconnects after a connection closes. (developer.mozilla.org)
If terminal access is appropriate, request your actual staging URL with:
curl --no-buffer --show-headers --max-time 20 \
https://staging.example.com/stream-check
Replace the example hostname. --no-buffer disables curl’s own output buffering; it does not disable buffering at your host. (curl.se)
3. Use arrival patterns to narrow the fault
Run the same test locally and through the deployed staging hostname. If your provider offers an authorized direct-origin testing method, compare that too. Do not expose the origin or weaken production access controls merely to bypass a CDN.
Interpret the following as diagnostic hypotheses, not proof:
| Observation | Next check |
|---|---|
| Timestamp messages arrive progressively, but AI output does not | Provider-event handling and backend forwarding |
| Earlier timestamps arrive together near the end | Application, compression or proxy buffering |
| A stream starts normally, then repeatedly stops at similar elapsed times | Function duration, proxy timeouts and application cancellation |
| curl receives progressive output, but the page displays everything at once | Browser parsing and rendering |
| Failures appear mainly with several tabs or simultaneous requests | Browser connections and host concurrency limits |
These hypotheses follow from the documented buffering, browser-body handling and duration mechanisms; confirm them with logs before changing configuration. (expressjs.com)
Worked interpretation: suppose your staging test produces timestamp strings one second apart, but the client displays all ten together. The timestamps show when the handler constructed the messages—not when they reached the browser. That pattern supports investigating buffering. It does not identify which layer buffered them.
For the real AI route, record a correlation ID and these milestones:
- Backend request accepted.
- Provider request started.
- First provider event received.
- First downstream body write attempted.
- Browser’s first parsed event.
- Completion or interruption.
Avoid logging message contents just to obtain timing evidence. Compare elapsed intervals within each process; do not treat unsynchronized server and browser clocks as precise latency measurements.
4. Fix buffering at the identified layer
If you control Nginx and the route uses HTTP proxying, investigate proxy_buffering. Nginx enables response buffering by default; disabling it passes data onward as received. The upstream response header X-Accel-Buffering: no can also control buffering, unless configuration ignores that header. It is not a universal hosting switch. (nginx.org)
For a PHP application reached through FastCGI, the relevant Nginx setting is fastcgi_buffering, not proxy_buffering. Verify which upstream mechanism your host actually uses. (nginx.org)
Compression is a separate checkpoint. Express’s compression middleware documents that SSE needs explicit flushing because compression buffers output. For staging, compare the affected route without compression, or use the middleware’s documented flushing method. Do not disable compression across the entire website as a first attempt. (expressjs.com)
Similarly, do not assume every CDN prevents streaming. Cloudflare Workers supports streamed response bodies, while code that reads the complete body loses that incremental behavior. Confirm the particular product, route and transformation involved. (developers.cloudflare.com)
Ask managed-host support a specific question:
“Does this route support incremental HTTP responses? Which component controls response buffering, and can it be changed for this route?”
After each approved staging change, rerun the timestamp test. Keep unrelated proxy settings unchanged.
5. Separate idle timeouts from total execution limits
“Timeout” is not one setting. Build a small inventory:
| Limit | What to establish |
|---|---|
| Connection establishment | How long the backend may take to connect upstream |
| Response inactivity | How long a layer tolerates no incoming data |
| Total invocation duration | How long the hosting function may run |
| Application deadline | When your own code cancels the operation |
Nginx’s proxy_read_timeout applies between successive upstream reads, not to the entire response duration. Therefore, a stream lasting longer than that configured interval is not automatically invalid if data continues arriving. (nginx.org)
A hosting function’s maximum duration is different. Vercel documents that it terminates functions exceeding their configured duration, including streamed-response waiting time. Sending periodic messages does not override that ceiling. (vercel.com)
SSE comment heartbeats can help prevent inactivity timeouts, but only where the relevant layer receives them. They do not make an indefinitely running request compatible with a finite invocation limit. (developer.mozilla.org)
For CDN failures, read the actual error documentation. Cloudflare’s Error 524 guidance distinguishes origin read and write timeouts and suggests status polling for long-running work. Do not diagnose every interrupted stream as the same timeout. (developers.cloudflare.com)
Hosting limits vary by runtime, plan and configuration. Verify the deployed function’s effective settings rather than copying an old limit from a tutorial.
6. Handle disconnects, completion and connection pressure
Do not label an answer complete merely because the network connection closed. OpenAI’s streaming guide provides separate completion and error events; your backend-to-browser protocol should likewise distinguish successful completion from interruption. (developers.openai.com)
Recommended application behavior:
- Show partial output as incomplete after an unexpected disconnect.
- Provide a visible Stop control.
- Attempt upstream cancellation when the user stops or disconnects.
- Clean up timers, listeners and other request resources.
- Avoid automatically starting another generation unless retry behavior is explicitly designed.
Fetch supports cancellation through AbortController. Server-side cancellation propagation depends on your runtime and provider SDK; verify it rather than assuming that closing the browser stops all upstream work. (developer.mozilla.org)
For a Node relay, respect backpressure: when response.write() returns false, pause further writes until the response drains or closes. The finite timestamp diagnostic is not a production relay implementation. (nodejs.org)
Test more than one open tab. MDN documents a low per-browser, per-domain SSE connection limit without HTTP/2; HTTP/2 uses negotiated stream limits instead. Also ask your host about simultaneous streaming requests and worker concurrency. (developer.mozilla.org)
Keep staging concurrency tests small and bounded. Record interruptions and completion, not just whether requests started.
7. Keep a fallback and a rollback path
A non-streaming mode is useful when incremental delivery is unsupported or unreliable: the backend waits for completion and returns one response. OpenAI documents both streamed and non-streamed response behavior. However, waiting for the full result still has to fit within your hosting and proxy deadlines. (developers.openai.com)
If work cannot reliably finish within one request, consider a durable job-and-polling design: return a job identifier, perform the work through supported background infrastructure, and let an authenticated client retrieve status and results. Cloudflare specifically recommends polling for large HTTP processes that hit its timeout boundary. (developers.cloudflare.com)
Do not silently retry a partially delivered answer. Instead, offer a clear retry choice, or retrieve an already-running job where your architecture supports it.
Before production rollout, use this acceptance checklist:
- Timestamp output arrives progressively through the intended path.
- Real AI output parses correctly and has explicit completion handling.
- Stop and disconnect behavior have been tested.
- Timeout and provider-error messages remain understandable.
- A small concurrent test completes without unexplained failures.
- The fallback works within verified limits.
- Previous application and proxy configurations remain available.
- Diagnostic routes are removed or remain restricted.
The objective is not to find a magic streaming header. It is to identify where incremental delivery stops, verify that layer’s supported behavior, and retain a usable alternative.
Read next: How to Benchmark Website Performance Before and After Adding an AI Feature