# Rush Hour on the GPU: Why Your AI Is 3x Slower at 3pm > Shared AI deployments are affordable because many customers' requests run on the same GPUs at the same time. That is also why the same request can be fast at midnight and run at under a third of the speed in the afternoon. A plain-language guide to concurrency, rush hour and noisy neighbors, and to when a dedicated deployment is worth paying for. - Author: Aleksander Pejcic (Co-founder & CTO) - Published: 2026-10-28 - Tags: inference, latency, deployments, multi-tenant - Source: https://sference.com/blog/rush-hour-on-the-gpu --- You type a question, and a moment later the answer starts streaming in. Most of the time it arrives faster than you can read it. Then one afternoon the same question, sent to the same model, comes back at a third of the speed. Nothing is broken. You have just met everyone else who is using that model at the same moment. This post explains, without the jargon, what happens behind an AI API, why its speed depends on how many people are using it, and when it is worth paying to have the hardware to yourself. ## What happens when you send a request AI models run on GPUs: specialized chips that each cost about as much as a car, usually installed eight to a server. A model like DeepSeek or Kimi is an enormous file of numbers that lives in those chips' memory. The model doesn't write in words. It writes in **tokens**, pieces of words that average about three-quarters of a word each. Answering you happens in two phases: 1. **Reading.** The model reads your whole prompt in one go. This is the pause before the first word appears, which the industry calls **time to first token**. A longer prompt takes longer to read. 2. **Writing.** The model produces the answer one token at a time. How fast that goes is measured in **tokens per second**. 100 tokens per second is about 75 words a second, roughly twenty times faster than most people read. It still adds up when an answer runs to thousands of tokens, or when a program rather than a person is waiting for the result. ## Shared deployments: one kitchen, many orders When you pay for an AI model per token, the way you do with most AI APIs, your request runs on a **shared deployment**: a set of GPUs serving one model to every customer at the same time. We call ours serverless; the idea is the same everywhere. Picture the GPUs as a kitchen with one very fast cook and a long griddle. The cook doesn't finish one order before starting the next. Instead, the cook makes pass after pass down the griddle, and on every pass adds one pancake to every order on it. On a GPU, one pass adds one token to every answer being written. The trick is that most of the work in a pass doesn't depend on how many orders there are. Before every pass, the cook has to re-read the whole recipe book; for a GPU, that means pulling the model's billions of numbers out of memory. Adding one more order to a pass is cheap by comparison, so a pass with thirty orders takes longer than a pass with three, but nowhere near ten times longer. That is why shared AI is affordable. Each pass serves many customers at once, so each one pays for a small slice of it instead of renting a whole server. ## Why your speed depends on everyone else Cheap isn't free. Every order on the griddle still needs its own pancake, so each pass takes longer as orders pile up, and your pancakes arrive more slowly. Here is what that looks like on DeepSeek V4.1 Flash, measured across every request it served on one day: ![Two charts side by side, both plotted against how many requests share the GPUs. Left: the median speed of one request falls from 343 tokens per second with about 4 requests sharing, to 217 with 11, 155 with 19, 129 with 27, 108 with 35, 98 with 43 and 87 with 54. Right: the total output of the GPUs rises over the same points to 1.8, 2.2, 2.6, 2.8, 3.1 and 3.4 times that of a quiet GPU.](https://sference.com/blog/rush-hour-on-the-gpu/assets/sharing-tradeoff.svg) Or drag the crowd onto the griddle yourself: ```sharing-slider Interactive widget on the website: drag how many requests share the GPUs and watch one request's speed fall while total output rises and cost per token falls. Same data as the chart above. ``` When only a handful of requests share the GPUs, each one streams at about 340 tokens per second, roughly 250 words a second. When around 50 share them, each one gets about 90, four times slower. Over the same range, the total output of the GPUs more than triples. That is the trade every shared AI service makes, ours included. **The more requests share a GPU, the cheaper each token gets, and the slower each answer arrives.** Every provider decides how far to push it. Push too far, and everyone's answers crawl. Don't push far enough, and everyone pays more. Our own answer is to keep the crowd small. We cap how many requests share a GPU, and when the kitchen is full we turn new work away immediately, with a retry-after, instead of letting every answer crawl. The wait for the first word stretches too. For a typical request it went from 0.19 seconds on a quiet GPU to 0.35 on a busy one. For the slowest one in ten, it went from 0.65 seconds to more than two. ## Rush hour You don't choose how many requests share the GPUs with yours. Everyone else using the model does, and they keep office hours. ![Two stacked charts covering one day, hour by hour in UTC. Top: the average number of requests sharing the GPUs stays low overnight, climbs from 08:00, and runs about four times higher from 09:00 to 15:00 before falling back by 16:00. Bottom: the median speed of one request is 330 to 390 tokens per second overnight, falls to between 112 and 163 from 09:00 to 15:00, and recovers to between 250 and 380 in the evening.](https://sference.com/blog/rush-hour-on-the-gpu/assets/rush-hour.svg) This is one ordinary Tuesday on the same model. Overnight in Europe, only a handful of requests share the GPUs and each one runs at 330 to 390 tokens per second. From late morning, when Europe is at its desks and the US East Coast is waking up, the kitchen is at its busiest and the typical speed drops to around 110 to 160. By evening it has recovered. Two things follow from this: - **Benchmarks disagree for honest reasons.** Test a model at 3 AM and you measure a quiet kitchen; test it at 3 PM and you measure rush hour. Both numbers are real, and neither is the whole story. - **Your own traffic counts too.** Fire fifty requests at once and you become part of the crowd you're measuring. Some of the slowdown you see is your own requests sharing the griddle with each other. ## Noisy neighbors The crowd isn't just a number. It's made of other people's requests, and some are far heavier than yours. Cloud computing calls this the **noisy neighbor** problem, after the neighbor whose party keeps you awake: it isn't happening in your apartment, but you can hear every bit of it. On an AI deployment, the loud neighbors look like this: - **A giant prompt.** Somebody sends a 300-page contract and asks for a summary. Before the model can answer, it has to read all of it, and that reading competes for the same chips that are writing your answer. In the kitchen, it's a wedding-cake order the cook has to work on between passes. Every pass slows down while the cake is in progress. - **A sudden burst.** Another customer starts a big test run and sends hundreds of requests in a minute. The crowd on the griddle doubles, and everyone's share shrinks. - **A very long answer.** A request writing tens of thousands of tokens holds its place on the griddle for minutes. You never see these neighbors. You only see some of your requests slowing down at random moments, which is why the unlucky requests suffer most when things get busy. In the numbers above, the typical first-word wait less than doubled as the GPUs filled up, while the slowest tenth more than tripled. Providers fight this, and it is where we spend much of our engineering time: capping how many requests share a GPU, [splitting giant prompts into chunks](/blog/prefill-head-of-line-blocking) so small requests can slip past them, and [predicting how long each request will wait](/blog/predicting-ttft) so it goes where there is room. That makes the noise smaller and rarer. It can't make it disappear, because on shared hardware, the sharing is the whole point. Our shared deployments also offer a [priority tier](/docs/guides/priority-tier), which moves your request to the front of the line when there is a queue. It doesn't make the kitchen any quieter once your order is on the griddle. Priority buys you a place in line, not speed. ## What shared deployments are great at None of this makes shared deployments a bad deal. For most workloads, most of the time, they are the right one: - **You pay only for the tokens you use.** There is no minimum spend, and quiet months cost nothing. - **There's nothing to set up.** The model is already running, and you can start sending requests straight away. - **Background work is cheaper still.** Requests that can wait a few minutes or hours go through our [flex and batch windows](/docs/pricing) at 30 to 50% off and run whenever there is room to spare. If your traffic is small, spiky or still taking shape, or if "fast most of the time" is good enough, stay shared. ## When you need a dedicated deployment Some products can't live with "fast most of the time": - **You've promised a number.** Your customers, or your own product team, expect something like "95% of answers start within a second" or "at least 100 tokens per second, all day". - **A person is waiting in real time.** Voice agents, live assistants, anything where a slow afternoon means a broken product. - **Your peak is everyone's peak.** If your users are busiest when everybody else's are, a shared deployment will be at its slowest exactly when you need it most. - **Your traffic runs around the clock** and is big enough to keep GPUs busy on its own. A **dedicated deployment** gives you GPUs reserved for you. It is the same model behind the same API, so your code barely changes: same model name, same request format, pointed at your deployment. What changes is who is in the kitchen. Just you. That fixes the problem at its root: - **No noisy neighbors.** Nobody else's wedding cake, nobody else's test run. - **Rush hour on the shared deployment doesn't reach you.** Your speed depends on your traffic and nobody else's. - **You choose the trade-off.** Fewer requests per GPU means faster answers; more requests per GPU means cheaper tokens. We size the deployment with you around your peak traffic and your speed target, and you pick a performance tier (cost saving, standard or turbo) instead of picking hardware. - **An SLA you can rely on.** Because we control everything that runs on those GPUs, we can put a speed and response-time target in writing. Custom SLAs are part of our enterprise agreements. One honest caveat: dedicated GPUs follow the same physics. Send ten times your planned peak and your own requests will crowd each other. The difference is that the crowd is yours. You can see it coming, plan for it, and give the deployment room to grow by setting how many copies of the model it can run. Dedicated capacity is billed per GPU-hour on a monthly commitment instead of per token. You pay for the kitchen whether it's busy or not, which is why it suits steady traffic: the busier you keep it, the less each token costs. ## How to choose - **"Usually fast" is good enough, and your traffic comes and goes?** [Start on shared](https://app.sference.com). No minimum, first call in minutes. - **The work can wait minutes or hours?** Shared, through [flex or batch](/docs/pricing), at 30 to 50% off. - **You need a guaranteed speed at 3 PM on a Tuesday, in writing?** [Talk to us about a dedicated deployment](mailto:hello@sference.com?subject=Dedicated%20deployment). - **Your traffic is steady enough to keep GPUs busy all day?** Dedicated, and it may well cost less too. Not sure which side you're on? [Tell us your peak and your target speed](mailto:hello@sference.com?subject=Size%20my%20deployment) and we'll size it with you, including telling you when shared is enough. If you want the best of both worlds, say so, we're cooking something for you. You can also read [how dedicated deployments work](/docs/deployments).