# Prompt caching

> How sference reuses shared prompt prefixes, what you can do to hit the cache more often, and how to verify hits.

Source: https://sference.com/docs/guides/prompt-caching

Requests that share an opening are the norm in production: an agent that resends the same system prompt and tool definitions on every step, a chat that replays its history each turn, a document a user asks several questions about. Paying full price to reprocess that shared text every time is waste, and on long agent loops it is usually the largest line on the bill.

sference matches the shared prefix for you and bills those tokens at the **cached input** rate, which is a fraction of the input rate on every model that offers one. Caching is **on by default**. There is nothing to enable. The rest of this page is about hitting it more often.

## Two rules decide whether a request hits

1. **A prefix matches byte for byte, from the very start of the prompt.** Matching stops at the first byte that differs.
2. **A cache is local to the worker that served the request.** A later request hits only if it lands on that same worker.

You control the first rule completely. The second is what `prompt_cache_key` is for.

## Rule 1: keep your prefix stable

Because matching starts at the beginning and stops at the first difference, the *order* of your prompt decides how much can be reused. Put what never changes at the front — system prompt, tool definitions, few-shot examples, a long document — and what varies at the end. Usually only the user's question changes, so it belongs last.

> **Dynamic content at the start kills your hit rate**
>
> A timestamp, a request id, or a session id in the opening line changes the first bytes on every request, so nothing behind it can be reused — including a system prompt that never changed at all. This is the single most common cause of a 0% cache hit rate.

```python ❌ Don't — the timestamp changes the first bytes, so nothing after it is reused
response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=f"Current time: {datetime.now()}\n\n{SYSTEM_PROMPT}",
    input=question,
)
```

```python ✅ Do — the prefix is identical every time; only the tail varies
response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=SYSTEM_PROMPT,
    input=f"{question}\n\nCurrent time: {datetime.now()}",
)
```

The same mistake wears other clothes:

- A request id or trace id interpolated into the system prompt.
- Tool definitions serialized in a different order between runs, or with object keys emitted in a different order. Serialize deterministically.
- Retrieved documents or a conversation summary placed *before* the system prompt instead of after it.
- A `datetime.now()` or `uuid4()` in a few-shot example.

## Rule 2: pin the worker with `prompt_cache_key`

A prefix cache lives on one worker. If your next request lands on a different worker, it re-prefills from scratch even though the text is identical.

`prompt_cache_key` solves this. Send a stable identifier for the sequence of requests that share a prefix — a chat session id, an agent run id, a document id — and sference routes every request with that key to the same worker while it stays warm.

```python
response = client.responses.create(
    model="zai-org/GLM-5.3",
    instructions=SYSTEM_PROMPT,
    input=user_turn,
    prompt_cache_key=session_id,   # same value for every turn of this session
)
```

```bash
curl https://api.sference.com/v1/chat/completions \
  -H "Authorization: Bearer $SFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [{"role": "user", "content": "..."}],
    "prompt_cache_key": "sess_8f14e45f"
  }'
```

`prompt_cache_key` is accepted on `/v1/chat/completions` and `/v1/responses`. Without it, placement is inferred from the input text, which works well when prefixes are long and distinctive but is best-effort.

> **Pick a key with the right lifetime**
>
> Use one key per conversation or agent run — not one per user, and not a global key. A key that is too broad spreads unrelated prefixes across one worker and wastes its cache; a key that changes every request defeats the purpose. If a session is idle long enough for the worker to be reassigned, the next request simply re-prefills once.

## What it costs

Cached input bills at the **cached input** rate listed on the [Pricing](https://sference.com/docs/pricing) page — typically 10–25% of the input rate depending on the model. There is **no cache-write charge**: unlike some providers, we do not bill a premium for populating the cache, and no TTL choice changes the price.

## Verifying hits

The token breakdown comes back on every response. On the OpenAI-shaped surfaces:

```json
{
  "usage": {
    "prompt_tokens": 8420,
    "completion_tokens": 118,
    "prompt_tokens_details": { "cached_tokens": 8192 }
  }
}
```

`prompt_tokens_details.cached_tokens` is the number of prompt tokens served from cache. If it is consistently `0` on requests you expect to hit, check, in order:

1. Is anything dynamic at the *start* of the prompt? (Rule 1 — the usual cause.)
2. Is `prompt_cache_key` set and stable across the requests? (Rule 2.)
3. Is the shared prefix long enough to be worth caching? Very short prefixes may not be cached at all.
4. Are consecutive requests arriving close together? A long idle gap can let the worker's cache evict.

On `/v1/messages` the same numbers appear as `cache_read_input_tokens` in the Anthropic-shaped usage block.

- [Pricing](https://sference.com/docs/pricing): Cached input rates per model, and how the windows stack.
- [Requests at scale](https://sference.com/docs/guides/requests-at-scale): Concurrency patterns that keep prefixes warm.
- [Agent SDKs](https://sference.com/docs/agents): Agent loops are the workloads that benefit most from caching.