·Blog

Engineering notes.

Async inference, batch scheduling, and building AI infrastructure outside the realtime assumption.

2026-10-27·Peter Us·Head of Inference Infrastructure

Tracking the KV Cache Across a Heterogeneous GPU Fleet

Our scheduler keeps a live index of what every engine in the fleet has cached, on GPU, in host memory, on disk and in caches shared between replicas. It is built from the engines' own cache events, and on a normal day it agrees with what the engines actually reuse on more than 99% of cached tokens.

inferenceschedulingkv-cache

2026-10-17·Aleksander Pejcic·Co-founder & CTO

We Benchmarked DeepSeek's V4.1 Flash Update: 20–30% Less Reasoning, Same Answers

DeepSeek's weights-only refresh of V4.1 Flash, measured before and after the cutover on the same lane and seeds: reasoning shrinks 20–30% on hard problems with correctness unchanged, and the reasoning_effort dial now works.

deepseekbenchmarksinferencemodels

2026-10-13·Peter Us·Head of Inference Infrastructure

Cache-Aware TTFT Prediction: Routing and Admission Under Load

Accepting every request past capacity makes the service slower for everyone. We predict each request's time to first token on every worker, from live engine cache events and prefill rates each worker learns on its own. We then use it to route requests and to turn away the ones we can't serve in time, while the client can still fall back elsewhere.

inferenceschedulinglatency

2026-09-22·Aleksander Pejcic·Co-founder & CTO

Prefill Concurrency in SGLang: Consistent TTFT Under Multi-Tenant Load

One oversized prompt can take the prefill stage hostage and stall every request behind it, including requests that have almost nothing left to do. We added a concurrent chunked prefill knob to SGLang's scheduler so the cheap requests get past, and sent it upstream.

inferencesglangschedulinglatencyopen-source

2026-08-03·Jernej Štrasner·Co-founder & CEO

Sference is live on Opper: European inference for frontier open models

Sference's managed inference is now available through the Opper gateway on a European route, with compute inside the EEA and zero data retention on by default. Kimi K3, GLM-5.2, DeepSeek V4 Flash, Qwen3.6 35B-A3B, Qwen3-VL 30B, and ThinkingCap are callable with an Opper API key today.

partnershipoppereuropeinference

2026-06-09·Aleksander Pejcic·Co-founder & CTO

Your Batch Pipeline Is Running Realtime Inference: The LET Trilemma

Prefect, Airflow, and Pydantic AI teach synchronous realtime inference patterns even inside batch DAGs, defeating the economic purpose of batch orchestration. We introduce the LET trilemma (latency, efficiency, throughput: pick two) and show how Sference reclaims batch economics.

inferencebatchefficiencyagentsarchitecture