Most teams reach for a smaller model first, which usually costs them quality they did not need to trade. These seven changes are free, take an afternoon, and several of them cut spend by more than switching model would.
When an inference bill gets uncomfortable the first instinct is to move to a smaller model. Sometimes that is right. More often there is spend being wasted in ways that have nothing to do with which model you picked, and fixing those costs you nothing in quality.
In rough order of how much they usually save.
1. Stop paying for context you resend every turn
The most common expensive mistake in a chat feature is sending the whole conversation on every request. Turn twenty costs twenty times turn one, and the user notices nothing.
Summarise older turns into a short running note and send that plus the last few messages. On a long conversation this alone can cut input tokens by an order of magnitude, and input is where the volume is in almost every real workload.
2. Check whether your model reasons, and whether you need it to
Models that reason before answering generate hidden tokens first, and you are billed for them. Of the 46 models we serve, 25 reason. The same one-line answer can cost a hundred times more on one of those than on an ordinary model.
If your task is classification, extraction, rewriting or a short reply, a non-reasoning model is almost always the right choice and the output is frequently indistinguishable. Save the reasoning models for tasks that genuinely need multiple steps.
3. Set max_tokens deliberately, in both directions
Too high is not free in every sense: it is what your reservation is priced against, so a large ceiling can push a small balance into a refusal for a request it could have afforded.
Too low is worse, and on a reasoning model it is a specific failure. The budget goes on the thinking
and you get an empty string with a full bill. We measured one model burning 303 credits at
max_tokens: 500 and returning nothing at all.
Set it a bit above the longest reply you actually want. You are billed on what is produced, not on the ceiling.
4. Move the format instruction out of the prompt
If you are asking for JSON by describing the shape in the prompt, you are paying for that description on every single request, and you are still getting fenced output some of the time.
Use response_format with a schema on a model that supports it. Fewer input tokens, no retries on
malformed output. Which models really support it is worth checking rather than assuming, and there is
a whole post on that.
5. Trim the system prompt
System prompts accumulate. Every instruction that survives from a debugging session six months ago is being paid for on every request forever.
Read yours. Delete anything you cannot point to a behaviour for. Then measure: half of what is in there is usually doing nothing you can detect.
6. Batch what does not need to be interactive
If you are classifying yesterday’s support tickets, you do not need one request per ticket. Put twenty in one request with a numbered output format, and you pay the system prompt once instead of twenty times.
The limit is context window and reliability, not cost. Past a certain batch size models start losing items, so test where yours degrades and stay under it.
7. Do not pay twice for the same question
If your users ask the same things, cache on a hash of the normalised prompt. A support bot answering “what are your opening hours” three hundred times a day should be making that request once.
Where you need the request to reach the model but must not be charged twice for a retry, send an
Idempotency-Key. A repeat with the same key replays the first response free rather than running
again, which matters most on exactly the flaky connections where your client is most likely to retry.
What we would not tell you to do
Do not switch to a model with a better tokenizer for your language without measuring per request. It sounds like the obvious optimisation and we have measured it going the wrong way by 6.4x, because the better-tokenizing model reasoned and the reasoning outweighed the saving.
Do not shorten prompts to the point of guessing. A low-cost wrong answer that a human has to redo costs more than the tokens you saved.
Measure before and after each of these on your own traffic. Every response we return carries its own cost, so the comparison is one query rather than a spreadsheet.