Rush hour on the GPU
One request streams fast at midnight and crawls at 3pm, because the griddle is shared. Drag the crowd onto it and watch the trade every shared AI service makes: your answer slows down while the GPUs serve more in total and each token gets cheaper to serve.
DRAG THE CROWD ONTO THE GRIDDLE
4
requests sharing the GPUs
343
tokens/s for your answer (~257 words/s)
1.0×
total output of the GPUs
1.00×
token cost
From the blog post Rush Hour on the GPU: Why Your AI Is 3x Slower at 3pm.