Prompt caching

Requests that share an opening are the norm in production: an agent that resends the same system prompt and tool definitions on every step, a chat that replays its history each turn, a document a user asks several questions about. Paying full price to reprocess that shared text every time is waste, and on long agent loops it is usually the largest line on the bill.

sference matches the shared prefix for you and bills those tokens at the cached input rate, which is a fraction of the input rate on every model that offers one. Caching is on by default. There is nothing to enable. The rest of this page is about hitting it more often.

Two rules decide whether a request hits

1. A prefix matches byte for byte, from the very start of the prompt. Matching stops at the first byte that differs.
2. A cache is local to the worker that served the request. A later request hits only if it lands on that same worker.

You control the first rule completely. The second is what prompt_cache_key is for.

Rule 1: keep your prefix stable

Because matching starts at the beginning and stops at the first difference, the order of your prompt decides how much can be reused. Put what never changes at the front — system prompt, tool definitions, few-shot examples, a long document — and what varies at the end. Usually only the user's question changes, so it belongs last.

response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=f"Current time: {datetime.now()}\n\n{SYSTEM_PROMPT}",
    input=question,
)

response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=SYSTEM_PROMPT,
    input=f"{question}\n\nCurrent time: {datetime.now()}",
)

The same mistake wears other clothes:

• A request id or trace id interpolated into the system prompt.
• Tool definitions serialized in a different order between runs, or with object keys emitted in a different order. Serialize deterministically.
• Retrieved documents or a conversation summary placed before the system prompt instead of after it.
• A datetime.now() or uuid4() in a few-shot example.

Rule 2: pin the worker with prompt_cache_key

A prefix cache lives on one worker. If your next request lands on a different worker, it re-prefills from scratch even though the text is identical.

prompt_cache_key solves this. Send a stable identifier for the sequence of requests that share a prefix — a chat session id, an agent run id, a document id — and sference routes every request with that key to the same worker while it stays warm.

response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=SYSTEM_PROMPT,
    input=user_turn,
    prompt_cache_key=session_id,   # same value for every turn of this session
)

curl https://api.sference.com/v1/chat/completions \
  -H "Authorization: Bearer $SFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [{"role": "user", "content": "..."}],
    "prompt_cache_key": "sess_8f14e45f"
  }'

prompt_cache_key is accepted on /v1/chat/completions and /v1/responses. Without it, placement is inferred from the input text, which works well when prefixes are long and distinctive but is best-effort.

What it costs

Cached input bills at the cached input rate listed on the Pricing page — typically 10–25% of the input rate depending on the model. There is no cache-write charge: unlike some providers, we do not bill a premium for populating the cache, and no TTL choice changes the price.

Verifying hits

The token breakdown comes back on every response. On the OpenAI-shaped surfaces:

{
  "usage": {
    "prompt_tokens": 8420,
    "completion_tokens": 118,
    "prompt_tokens_details": { "cached_tokens": 8192 }
  }
}

prompt_tokens_details.cached_tokens is the number of prompt tokens served from cache. If it is consistently 0 on requests you expect to hit, check, in order:

1. Is anything dynamic at the start of the prompt? (Rule 1 — the usual cause.)
2. Is prompt_cache_key set and stable across the requests? (Rule 2.)
3. Is the shared prefix long enough to be worth caching? Very short prefixes may not be cached at all.
4. Are consecutive requests arriving close together? A long idle gap can let the worker's cache evict.

On /v1/messages the same numbers appear as cache_read_input_tokens in the Anthropic-shaped usage block.