# Tracking the KV Cache Across a Heterogeneous GPU Fleet > Our scheduler keeps a live index of what every engine in the fleet has cached, on GPU, in host memory, on disk and in caches shared between replicas. It is built from the engines' own cache events, and on a normal day it agrees with what the engines actually reuse on more than 99% of cached tokens. - Author: Peter Us (Head of Inference Infrastructure) - Published: 2026-10-27 - Tags: inference, scheduling, kv-cache - Source: https://sference.com/blog/kv-cache-index --- In our [post on predicting time to first token](https://sference.com/blog/predicting-ttft) we left one thing for later: how the scheduler knows what each engine has cached, including offloaded and shared caches. That's this post. Serving a request has two stages. Prefill reads the whole prompt and writes its KV cache, the attention keys and values for every token. Decode then generates the answer one token at a time, reading that cache at every step (Aleksander covers both in [his post on prefill concurrency](https://sference.com/blog/prefill-head-of-line-blocking)). Reusing the KV cache is the biggest lever there is for prefill. A prompt whose prefix is already cached skips most of its prefill, and in multi-turn conversations, agents, or requests that share a long system prompt, most of a prompt usually is. It is cheaper for the client too: cached tokens are billed at a [fraction of the input rate](https://sference.com/docs/guides/prompt-caching). One way to get more reuse is to share the cache between replicas, so a prefix computed on one can be loaded by another. DeepSeek recently [described](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md) taking this all the way: the KV cache for their whole fleet sits on one dedicated storage cluster, any machine can hit any prefix, and they route on load alone, which keeps cache affinity from piling requests onto a few replicas. That works when the whole cluster is purpose-built for serving, down to the hardware: the storage, the network, the machines. Our fleet is more varied: different engines, models, GPU vendors and generations, each replica with its own mix of cache tiers, from GPU and host memory to local disk and shared pools. Reusing a prefix costs very different amounts depending on where it sits. So our scheduling layer, the part of the platform that picks a replica for every request, needs to know how much of a prompt is cached, and where. It can get that wrong in two ways: - It thinks a replica is colder than it is. It routes the request somewhere worse, or turns it away when it would have been served in time. - It thinks a replica is warmer than it is. It admits a request that then misses its latency bound. Either way, the fleet also does work it didn't need to: a prefill recomputed while a cached copy sat on another replica, or a prefix pulled back from a slower tier while a GPU copy existed. And the client pays for it: a prefix that was cached somewhere but missed is billed at the full input rate instead of the cached one. ## How the index works LLM engines don't cache a prompt as a whole. They split its KV cache into blocks of a fixed, configurable number of tokens, and each block is identified by its tokens together with everything before it. Two prompts that start the same way share every full block up to where they first differ, and a block can only be reused if every block before it is cached too. Cache eviction can mean two different things. Blocks can be deleted, usually from the end of a prefix, and the next request recomputes that part. Or they can be moved down to host memory or disk, where they are cheaper to load back than to recompute, but not free. Which blocks went, and where, is something only the engine knows. So we don't guess. Most serving engines and KV cache systems, vLLM, SGLang, TensorRT-LLM and LMCache among them, can publish an event whenever a block of KV cache is stored or removed. Next to every engine we run a relay that subscribes to those events and streams them to the scheduler, which keeps an index of the whole fleet. The index is a tree of blocks for each model, shared by all the replicas that serve it. Each block in the tree records where it currently lives: on which replicas, in which tier, and with data-parallel attention on which rank, since each rank keeps its own cache. To price a request, the scheduler walks the prompt down the tree, and how far it gets on each replica is how much of the prompt that replica has cached. ![Engines publish cache events through a relay to the scheduling layer's KV cache index; incoming requests are routed back to the best replica, or rejected early](https://sference.com/blog/kv-cache-index/assets/kv-event-flow.svg) The relay also unifies the engines' formats, keeps track of which tiers hold each block, and can replay everything the engine holds when the index needs to rebuild. When it can't tell whether a block is still reusable, because of quirks in some engine and model combinations or with some hybrid models (more on those below), it reports it as gone: treating a warm replica as cold costs us a little, treating a cold one as warm costs the client a slow request. ## One index, many engines Most LLM engines and KV cache systems report their cache the same way: blocks were stored, blocks were removed, or the whole cache was cleared. Many of them share a lineage, and that common shape is what lets one index cover a fleet that mixes them. The shape is shared; the meaning isn't, quite: - **Eviction.** In some engines, removing a block means everything after it is gone too; in others, later blocks can stay cached. The index follows each engine's rules. - **Hashes.** Every engine hashes blocks its own way, so the same prefix looks different on different replicas. The index hashes blocks itself, the same way everywhere, and uses the engine's hash only to match evictions to blocks. - **Speculative decoding.** With EAGLE-style drafting, some engines key the cache on pairs of tokens, because the draft model's KV at a position also depends on the next token. The relay turns them back into single tokens. - **Versions.** There is no official standard for these events (yet), so their details can change with any engine release. We pin the versions we run and check the event stream on every upgrade. ## Beyond GPU memory The index covers all the places a prefix can live, and the combinations of them we run: - **GPU memory**, the fastest and the smallest. - **Host memory and local disk.** Engines and cache systems can keep a block evicted from GPU in host memory or on disk, where it stays reusable. - **Caches shared between replicas.** Host memory shared by the replicas on a node, or a pool shared across nodes, so a prefix written by one replica is warm for the others. Sharing is independent of offloading. A replica can offload to its own host memory and disk without sharing anything, or read a shared pool with no local offload at all, and the index handles any combination. ![Two nodes: on the first, two replicas have their own GPU memory and share host memory and local disk; the second has one replica with its own tiers; separately, a cache shared between nodes that any replica can read](https://sference.com/blog/kv-cache-index/assets/kv-cache-tiers.svg) A move between tiers is an eviction from one tier and a store in another. The scheduler has to tell that apart from the block being gone. The block is still cached, it is just more expensive to reuse, and the scheduler takes that into account when it compares replicas. ## The hard parts The index is a copy of state that lives somewhere else, and keeping a copy honest has the usual problems. - **Lag.** Events arrive after the fact. A prefix that is being prefilled right now isn't in the index until the engine stores it, and an eviction isn't visible until its event arrives. A request routed in that window is priced on a slightly old view. We narrow the window by assuming a prompt is cached on the replica we just sent it to, before the engine confirms it, so the next turn of a conversation or a burst of requests with the same prefix already counts it as warm. - **Races.** Under load, a block can be evicted between the moment we route a request and the moment the engine gets to it: the index was right when it answered and wrong by the time it mattered. A move between tiers also arrives as two events, a removal and a store. The relay tracks every tier a block is in and retracts it only when the last one drops it, so the index never sees a move as a loss. - **Gaps and rebuilds.** If the stream drops events, or a new version of the scheduler needs to rebuild its index, the index has to be brought back in sync. The relay keeps a mirror of its engine's cache and replays it as a snapshot, shallowest blocks first, since those are the prefixes most prompts share. Routing gets the valuable part right away while the rest backfills, and the replays are spread out so the replicas don't all send at once. - **Memory and volume.** The index holds every cached block of every replica in every tier, and host memory and disk hold far more than GPU memory. Block size is a trade-off: smaller blocks mean more events and a bigger index, larger ones mean coarser reuse, since only full blocks are reused and two prompts that part ways mid-block can't share that block. We run engines with blocks of 64 to 512 tokens, depending on the model. And the index keeps only block hashes, never the prompts. Hybrid models are the hardest case. More and more new models mix attention with linear attention or state-space layers: Kimi-K3, GLM-5.3-Flash, Qwen3.8 and others. Those layers don't keep a cache per token, only a fixed-size state that sums up everything before it. To resume a prompt at some point, the engine needs both the KV up to that point and a copy of the state from exactly there. That complicates the index in three ways: - **Checkpoints.** A state copy is large, so engines keep one only at some points along the prompt. Reuse stops at the last one, even if the KV after it is still cached. - **Separate eviction.** State copies are evicted on their own. A prefix can lose its state while its KV is still cached, and with it the point the engine could resume from. - **Missing events.** Not every engine reports it when only a state copy is evicted. There the index still sees the KV, so what it credits is an upper bound on what the engine will actually reuse. Models with sliding-window attention, like DeepSeek-V4 and gpt-oss, have a milder version of the same problem: the window is cached separately, and if it's gone the end of the prefix has to be recomputed. Between these and the differences between engines, a perfectly accurate index is out of reach. But it can get close, and we measure how close. ## How close we are Every request tells us how close the index is. When we send a request to a replica, we record how much of it we believe is cached. When it finishes, the engine reports how much it actually reused. Over a typical day: - the engines reused 93.1% of all prompt tokens from cache, - the index had credited 99.6% of the tokens the engines reused, - and it credited only 0.3% that the engines didn't have. ![Per model, the engines served 91 to 97% of prompt tokens from cache, the index had credited 99.1 to 99.97% of the tokens the engines reused, and it credited under 1% that the engines didn't have](https://sference.com/blog/kv-cache-index/assets/kv-index-agreement.svg) On every model, the index catches almost all of the reuse, and even its biggest overshoot, on the hybrid GLM-5.3-Flash, is under 1%. ## What's next As more models go hybrid, keeping the index this close gets harder. Better events from the engines will help, and every request already tells us what the engine actually reused, so we plan to feed that back: when an engine reuses far less than we credited, stop crediting that prefix there. And as more of the fleet runs with host, disk and shared caches, we keep refining how we weigh a prefix in GPU memory against one it would have to load back. So far the index tells us where a prefix is. The next step is deciding where it should be. While a request waits in a queue, the replica it is going to could already be loading its prefix from host memory, disk or a shared cache, so the load overlaps with the wait instead of adding to it. And a prefix a session keeps coming back to could be copied ahead of time to a replica with room, before the replica that holds it gets busy. The same goes for eviction. Engines mostly evict by recency or another local policy, because they only see their own requests. The scheduler sees more: which sessions are still active, which prefixes are shared by many prompts, which kind of traffic tends to come back and which doesn't. That's enough to predict which prefixes are worth keeping on GPU, which can move down a tier, and which can go entirely.