Streaming, retries and timeouts all behave differently when the network drops halfway through a two-second response. Most LLM tutorials assume a connection that does not. Here is what actually breaks and what to do about each one.
Almost every guide to building with an LLM assumes a stable connection. If your users are on mobile data that drops in a lift, a lot of that advice quietly does not hold, and the failures are the expensive kind: charges with no output, duplicate charges, and a UI stuck forever on a spinner.
This is what we hit building on exactly that network, and what to do about each one.
A disconnect is not a free request
The first thing to internalise: when your user’s connection drops mid-stream, the model does not stop generating. It finishes, the provider bills for the whole thing, and nobody is left holding the output.
So the question is never “how do I avoid paying for an abandoned request”, it is “how do I make sure the accounting is still correct when nobody is listening”. If your own metering happens after you finish writing to the client, it will not run: with no reader draining the response, the write blocks on backpressure rather than throwing, your cleanup code is never reached, and the runtime eventually kills the whole thing.
We measured this on our own path before we fixed it: an aborted stream reserved 18 credits and refunded 0. It looked like an overcharge bug and it was really a control-flow bug.
The fix is to split the stream in two. One branch goes to the client. The other is drained purely for accounting and never awaits anything the client controls, so it completes whatever the client does. If you are proxying a model yourself, this is the single most important shape to get right.
Retries need an idempotency key, not just a retry
On a flaky connection your client will retry. If the first request actually reached the model and only the response was lost, a naive retry pays twice.
Send a unique key with each logical request and have the server refuse a duplicate rather than
re-running it. We honour Idempotency-Key for this: the same key with the same body replays the first
response for free, a different body is a 422, and a request still in flight is a 409 rather than a
second charge.
One warning if you are implementing this yourself: do not store the claim in an eventually-consistent cache. We wrote it against KV first and it measurably did not work. Two simultaneous retries both read “no claim exists”, both proceeded, and both were billed 27 credits. Moving the claim to a conditional insert in a real database, so the database picks the winner, fixed it: three concurrent requests, one served, two refused, charged once.
Distinguish “slow” from “dead” honestly
A model can take fifteen seconds. A dead connection also takes fifteen seconds. Your timeout cannot tell them apart, but a stream can.
Time to the first token, not to completion. If nothing has arrived in eight seconds something is wrong; if tokens are still arriving, keep waiting however long it takes. This single change removes most spurious timeouts, and it is only possible if you stream.
Stream even when you render the reply all at once. The first token is your liveness signal.
A mid-stream failure will not arrive as a status code
Once the response headers are sent there is no status code left to fail with, so an upstream failure arrives as a data frame in the stream instead.
Before we handled this, a caller got a perfectly well-formed but empty or truncated stream that ended in a normal stop reason, was billed for what had been generated, and had no way to tell it apart from a model that finished early on its own. Check for an error frame and surface it as an error; a truncated answer presented as a complete one is worse than a visible failure.
Related: keep-alive comments are interleaved into SSE by several providers to stop proxies timing out a slow stream. They are not JSON. Guard on the frame prefix rather than on blank lines, or your parser will throw on a comment.
Make the UI reflect three states, not two
A request is not either done or failed. There is a middle state that lasts real seconds, and pretending otherwise is what makes an app feel broken on a slow network.
Show it. A caret that streams tokens as they arrive turns a fifteen-second wait into something legible. And if you are showing what a request cost, do not compute it yourself from the stream to fill the gap: your count is an estimate and the charge is not, and a number that differs from the invoice is worse than a moment of “still metering”.
Design for the pre-JavaScript render
Last one, and it is the one most often missed. If your feature is a client-side component that fetches on mount, a user tapping before your bundle arrives gets nothing at all.
Server-render whatever you can: the model list, the prices, the empty state. Hydrate for the interactive part. On a fast connection nobody notices the difference. On a slow one it is the difference between a usable page and a blank one, and those users are not an edge case in this market.