You built your pipeline on Prefect, Airflow, Dagster, or Temporal because the workload tolerates delay. A document processed at 3 AM is worth the same as one at 3 PM. So you right-sized your CPU cluster, ran it at 80% utilization, and kept costs predictable by spreading work across time instead of provisioning for peak.
Then you added an LLM agent, just a simple one line in your DAG. The framework called a realtime API under the hood and you had no reason to think that behaved any differently from the batch execution you were already running. Your dashboard stayed green. Retries worked as before. But underneath, your carefully sized batch compute was now riding on someone else's mostly idle GPU cluster, and the economics there are different enough that your cost model broke without anyone noticing.
What the frameworks do under the hood
Prefect & Marvin, Dagster, Temporal, and Pydantic AI all route LLM tasks to realtime inference endpoints. Temporal's own AI cookbook is explicit about it: the recommended pattern wraps a synchronous client.responses.create(...) call in an activity, one request, one realtime response, with retries moved to the workflow engine. None of them use cheaper batch inference by default as a native DAG operation. To the scheduler, the LLM call is just another Python function, instead of a group→map→collect that batches requests, fans them out concurrently, and gathers the results — the very pattern these frameworks already expose for every other task.
Airflow is the only one to provide a batch operator example via OpenAITriggerBatchOperator which uploads a JSONL file of requests and submits it to OpenAI's async /v1/batches endpoint, then waits for completion in deferrable mode.
The framework still does its real job well, orchestrating retries and DAG dependencies on your CPU nodes. The inference itself, though, runs on someone else's GPU cluster through an API built for chatbots. You've quietly outsourced the compute-efficiency problem, and your budget leaks out through that hole.
The whole ecosystem behaves this way, for example, the "async" in Pydantic AI means non-blocking I/O against a realtime endpoint. It does not mean accumulating requests for a cheaper batch job.
Agent harnesses like OpenClaw, PI, and Claude Code make the same realtime assumption. What a genuinely async-native harness would look like is a separate post.
The utilization gap no one shows you
On your own CPU infrastructure, 20% average utilization is a problem. Your CTO sees it, asks questions, and you go right-size.
GPU inference plays by different rules. Cluster utilization across the industry sits around 20–30%, and some deployments run as low as 5% (ChipOnMyShoulder). A good chunk of the supply crunch everyone talks about (Sequoia Capital) comes down to chips we already own sitting idle most of the day. The most expensive GPU in your stack is the one you've already paid for that does nothing.
None of that shows up on your CloudWatch dashboard. The GPUs sit behind a realtime API, and the provider folds the cost of their idleness into per-token pricing. You pay the token rate and assume someone upstream optimized it.
They optimized for latency, because that is what chatbot traffic needs. Your batch job inherits the bill for a guarantee it never uses.
From CAP to LET: the inference trilemma
Database people know CAP. When a network partition hits, you choose consistency or availability and live without the third. Inference has its own version across three axes: latency, efficiency, throughput. Pick two. We call it the LET trilemma.
It holds because low latency and high efficiency pull in opposite directions. Whatever you do to get one, you spend the other.
- Latency + throughput, paying in efficiency. This is the realtime corner. Holding TTFT under a few hundred milliseconds for many concurrent users means weights stay permanently loaded in HBM and every node carries spare compute headroom. Fill every SM and the next request queues, so TTFT spikes. You provision for peak, then watch those GPUs draw full power at 3 AM amortized over a fraction of the daytime tokens.
- Latency + efficiency, paying in throughput. A small dedicated deployment: one model, one warm node, kept busy by a steady trickle. Fast and well-utilized, until a spike arrives that a single node can't absorb. You've capped how much work you can take.
- Throughput + efficiency, paying in latency. The batch corner. Accumulate requests, reorder and pack them, size capacity to the queue, switch GPUs off when it drains. You give up "answer in 200 ms" and get utilization and throughput in return.
Every corner sacrifices the axis it can't reach. Realtime providers sit in the first one because their customers demand it. Batch workloads belong in the third. The frameworks route them to the first regardless.
How Sference reclaims batch economics
Sference ships an SDK that fits natively into your batch DAG (Prefect batch example). In Prefect, Airflow, or Dagster it's a first-class operation, not a Python function that happens to call an API, so your pipeline hands us the whole group→map→collect job instead of one blocking request at a time.
We are building a batch-first inference stack from the ground up.
- Accumulate requests and schedule them when capacity is cheapest — the inference equivalent of running flexible load during off-peak hours.
- Group by model and prompt prefix to maximize cache reuse over long a time window.
- Load model A, drain its queue, evict it, load model B. We treat weights as a cache, not permanent residents.
- Match GPU capacity to actual demand. Grow the cluster only when the queue deepens and maximize for efficiency, not just for a daily volume spike.
- Right-size the model on the right-size GPU. We place each model on the cheapest hardware that serves it well.
- Right-size the model for the task. Classification, extraction, and embedding jobs rarely need a frontier model, a smaller open-weights model at batch rates often delivers the same accuracy at a fraction of the cost.
The result is utilization a realtime SLA simply can't reach, because we provision for the queue we actually have instead of the peak we might see. Customers see that as a lower cost per token.
The check you should run
If you charge your EV overnight because off-peak electricity is cheaper, why pay peak-provisioned token rates for work that could wait until 3 AM?
Here's the five-minute check. Pull last month's LLM bill and split the calls into two buckets: a human was waiting on the response, or nobody was. The enrichment job, the nightly classification run, the embedding backfill all go in bucket two. Now multiply it by the gap between realtime and batch pricing. That number is what the realtime default costs you every month.
You already treat compute as a variable-cost resource on your side of the DAG. The inference call is the exception, and the realtime API won't let you shift the load. It stays on, stays warm, and stays priced for peak, because the provider can't power it down without breaking someone else's SLA. You end up paying for a latency guarantee your batch job never asked for.
If that number made you wince, run the Prefect example.