Prompt caching

A long system prompt sent on every turn is mostly identical every time. Our supplier caches matching prefixes automatically, so you already get the latency benefit with no change to your code.

What is reported

Every response carries the count:

"usage": {
  "prompt_tokens": 2048,
  "completion_tokens": 31,
  "total_tokens": 2079,
  "prompt_tokens_details": { "cached_tokens": 1920 }
}

Raising the hit rate

Pass prompt_cache_key. It keeps a conversation’s turns on the same cache lineage, which matters for multi-turn chat where the shared prefix is the whole system prompt plus the history so far.

client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages,
    prompt_cache_key=f"conversation-{conversation_id}",
)

Use a value that is stable for a conversation and different between conversations. A user id works; a per-request uuid defeats the point.

What helps

• Put the stable part first. Caching matches on prefixes, so a system prompt followed by history caches well and a timestamp at the top of every message caches nothing.
• Do not reorder. A tool list shuffled between calls is a different prefix.
• Longer prefixes gain more. A fifty-token system prompt is not worth designing around.