Sference runs frontier open models on shared deployments, and the two things we get judged on are speed and consistency. Your requests land on the same GPUs as everyone else's, which is what makes the economics work and is also the entire difficulty: one customer's traffic cannot be allowed to turn into another customer's latency.
We do many kinds of optimizations. Fusing CUDA kernels, to take one example, buys around 5% on TTFT, and we will take that when it is sitting there. What we have found is that this class of work is not what moves the numbers a customer actually feels. The KPIs they watch, speed and consistency, are set much further up the stack: which requests get admitted, in what order, and what is allowed to share a slot with what. Architectural choices come back in multiples rather than percentages, so that is where most of our effort goes.
This post explains how we implemented prefill concurrency in SGLang, and how it improves TTFT in a multi-tenant environment for all our users.
Understanding a single request
How a request actually gets served
Serving a request happens in two phases, and they stress completely different parts of the GPU.
Prefill reads the prompt. The whole thing goes through the model in a single pass, with every token attending to every other token, which comes out as dense matrix multiplication. It keeps the GPU's arithmetic units busy and finishes about as fast as the chip can do math. What it produces is the KV cache: one key and one value vector per prompt token, written once. Prefill is what you are waiting on when you wait for the first token, so it sets time to first token (TTFT).
Decode writes the answer, one token at a time. Every new token has to look at the entire KV cache, so each step pulls that whole cache out of GPU memory to produce a single vector. That is a bandwidth problem rather than a math problem, and the arithmetic units sit mostly idle waiting on reads. Decode sets inter-token latency (ITL), which is why a long answer can feel slow even when nothing at all is wrong.
The distinction matters because of how the two queue. Decode steps are small and roughly the same size as each other, so a scheduler can interleave them without much trouble. Prefill arrives as one lump of work whose size is chosen by whoever sent the prompt. That is where a scheduler gets stuck.
TTFT = queue wait + prefill
ITL = one decode stepWhen someone complains that the model "took forever to start answering", they are describing TTFT. On a healthy system that number is mostly prefill. On an unhealthy one it is mostly queue wait.
Why the median looks fine
Prefix caching changes the shape of the workload by reusing the KV cache that prefill already built for an earlier request. If your traffic has any structure to it, a long system prompt, a shared document, a conversation that keeps going, then most requests hit the cache and only pay for the tokens that are genuinely new.
So the distribution of work becomes bimodal. A large mass of requests have a few hundred new tokens and finish prefill almost immediately. A small number have hundreds of thousands and take seconds. The median sits comfortably in the first group, which is why it looks so good. Everything that breaks the product is in the second group, along with everyone unlucky enough to arrive behind them.
Here is what prefill actually costs on our own traffic:
Multi-tenant architecture
One slot, one request
Up to here this has been one request at a time: what prefill costs, and what makes a prompt expensive. None of this explains why your small prompt with a high prefix cache hit can feel slow.
SGLang's unified scheduler, like most serving stacks, admits prefill work against a token budget. In the stock behavior a request is admitted whole: it takes as much of the budget as it needs, and it holds that budget until it is finished.
Take the "before" panel. A cold prompt arrives with 600,000 uncached tokens. It gets admitted and owns the prefill slot for the next 105 seconds. Six more requests arrive behind it: four small ones, and two long prompts with a 99% cache hit that have almost nothing left to prefill. All six wait, because the slot is occupied. All six get their first token at the 105-second mark.
That is head-of-line blocking. The scheduler has no way to express "this request is nearly free", so a 300,000-token prompt at a 99% hit rate queues exactly like a 2,000-token one, and prompt length turns out to have very little to do with who waits.
It comes down to one line of accounting in PrefillAdder (sgl-project/sglang). The scheduler tracks its remaining token budget and decrements it for every request it admits:
self.rem_chunk_tokens -= extend_input_len # fires for every admitted requestOnce a long request is in, the remaining budget is zero and nothing else can be scheduled alongside it. The pipeline is now one request wide.
The fix
We added a per-request cap on how many tokens a single prefill may take in one scheduler round, called --long-prefill-token-threshold after the equivalent flag in vLLM's V1 engine. It runs on our own fleet, and the patch is open upstream as sgl-project/sglang#34623 against the v0.5.18 scheduler.
That turns one number into two:
chunked_prefill_sizeis the total token budget for a round. Think of it as the batch.long_prefill_token_thresholdis the most any single request may take out of that budget.
Divide the first by the second and you get how many partial prefills the scheduler will run at once. On our Kimi-K3 deployment, 8192 // 4096 gives two. On DeepSeek-V4-Flash, 16384 // 4096 gives four.
A request longer than the threshold is admitted as a chunked request. It gets its threshold worth of tokens this round, the rest of the budget goes to whoever else fits, and it comes back for the next chunk in the next round. Anything shorter than the threshold is still admitted whole, so the common case is untouched.
The "after" panel shows the same 600,000-token prompt served in chunks. The small requests get served in the gaps and see their first token in under a second instead of 105. The two high-hit long prompts get their 3,000 tokens in one gap and finish at 1.16s and 2.37s. The cold prompt still gets all of its work done, it just stops being the only thing running.
If you serve mixed traffic, long documents next to short agent calls, or anything at all with prefix caching in front of it, this is worth testing on your own workload. The flag is --long-prefill-token-threshold, and a reasonable starting number is your chunked_prefill_size divided by the concurrency you actually want.
What the fix costs
The obvious cost is that the cold prompt gets slower. In the scenario above it finishes at 112 seconds instead of 105, because it is now sharing the slot with the requests it used to block. Roughly 7 seconds taken from the big request bought back around 100 seconds for the six behind it. That is a good trade if your traffic is mixed and a bad one if it isn't.
Two other things to know before you turn it on:
- Uniform workloads get worse. If every request in your queue is a 500,000-token prompt, first-come-first-served is already the right answer and chunking only adds scheduling overhead and context switches. The win comes entirely from heterogeneity. Without it, leave the threshold at zero.
- A long prefill with nothing to share the budget still pays for chunking. With no other work around to fill the remaining budget, a lone long request gets capped at the threshold anyway and the rest of the budget idles while it waits for its next round. Set the threshold against the concurrency you actually have, not as low as it will go.
One thing that is not a cost: the threshold is KV-neutral. It changes the order in which work is done, not how much KV cache any request needs. Token budget is what drives KV pressure, and that is the knob to reach for when you are memory-bound.
You will run out of KV capacity eventually, and with 600,000-token prompts in your traffic you will run out sooner than that. The answer is not to shrink the budget until prefill is slow again, it is to add a tier. We run SGLang's HiCache, which extends the KV cache out of the device pool and into host memory, so long prefixes survive on a node that could never hold them in GPU memory alone. The two changes work together. HiCache is what makes it viable to keep a 600K-token prefix around at all, and the threshold is what stops building one from starving everything else while it happens.
What comes next
The threshold is a ceiling today, and that is the part of the design we are least happy with. chunked_prefill_size gets carved into fixed slices of long_prefill_token_threshold, one per concurrent prefill, and no request may take more than its own slice. When the concurrency is genuinely there, the whole budget gets used. When it isn't, the unclaimed slices sit empty and the long request crawls along at its cap for no reason, which is the second caveat above.
The better shape is a floor instead of a ceiling. The threshold would guarantee a long request at least that many tokens per round, which is all you need to keep short prompts slipping past it, and with nothing else competing it would expand into whatever is left of chunked_prefill_size. Concurrency would then decide the split at runtime rather than at configuration time: two long prefills take half the budget each, one on its own takes all of it, and the flag stops being a number you have to tune against your traffic mix. That is not in the patch we shipped. It is where we would like to take this next.
None of this is really about SGLang. Any scheduler that admits prefill work in indivisible units will eventually hand your whole deployment to whoever sends the largest prompt, and the reason it stays invisible is that the median is looking somewhere else while it happens. If you haven't checked your own p99 lately, that is where it would show up.