Requests at scale
Move tens of thousands of requests through sference without serializing them one by one.
Requests at scale
A loop that fires requests sequentially stalls as soon as the workload grows past a few hundred calls. sference gives you two ways to clear large volumes: fan out concurrent background responses, or hand a single bundle to the Batch API and let sference work through it.
Which path fits your workload
Concurrent background responses
You want results as each one finishes, need per-request control, or care about latency on individual items. You manage concurrency.
Batch API
You have a fixed set of requests and would rather submit once and collect later. sference owns the queue.
| Background responses | Batch API | |
|---|---|---|
| Unit of work | one HTTP request each | one JSONL file |
| Results | poll each response id, or webhook | poll the batch, download JSONL |
| Concurrency | you bound it | sference schedules it |
| Per-item control | full — cancel, retry, inspect individually | limited to the batch |
| Best for | dynamic workloads, agent fan-out, streaming results | fixed datasets, evals, offline jobs |
| Window | 24h batch | 24h batch |
The running example below is sentiment scoring for a backlog of product reviews.
Fan out with background responses
Submit each request with background: true and keep many in flight. For workloads of roughly 1,000 requests and up, four practices keep it stable:
- Use
AsyncOpenAIwithDefaultAioHttpClient(). Under high concurrency the aiohttp backend outperforms the defaulthttpxtransport. - Bound concurrency with an
asyncio.Semaphore. A fixed limit (200 is a sound starting point) caps simultaneous connections so you never exhaust them. - Submit all requests first, then poll in a second pass. Separating submission from polling keeps both phases simple and avoids interleaving.
- Set
max_queue_wait_sso a congested moment fails fast instead of holding connections open — see Max queue wait.
pip install tqdm 'openai[aiohttp]'import asyncio
from tqdm import tqdm
from openai import AsyncOpenAI, DefaultAioHttpClient
CONCURRENCY = 200
async def score(client, review: str) -> str:
response = await client.responses.create(
model="zai-org/GLM-5.3",
input=f"Reply with one word, positive or negative:\n\n{review}",
background=True,
max_output_tokens=16,
max_queue_wait_s=30,
)
return response.id
async def main(reviews: list[str]):
async with AsyncOpenAI(
base_url="https://api.sference.com/v1",
api_key=SFERENCE_KEY,
http_client=DefaultAioHttpClient(),
) as client:
sem = asyncio.Semaphore(CONCURRENCY)
async def submit(review):
async with sem:
return await score(client, review)
# Pass 1: submit everything, collect ids
ids = []
for coro in tqdm(asyncio.as_completed([submit(r) for r in reviews]), total=len(reviews)):
ids.append(await coro)
# Pass 2: poll to completion
results = {}
pending = set(ids)
while pending:
for rid in list(pending):
r = await client.responses.retrieve(rid)
if r.status in ("completed", "failed", "incomplete"):
results[rid] = r
pending.discard(rid)
if pending:
await asyncio.sleep(1)
return results
asyncio.run(main(REVIEWS))Collect on the finish reason, not on status alone
On a reasoning model, max_output_tokens covers hidden reasoning as well as the answer. A response can settle incomplete with a perfectly good answer attached. Check the incomplete detail and use the output if it is there, rather than discarding it.
Prefer webhooks to polling
Polling every response id in a loop is the simplest thing that works, but it is wasteful at scale. If your workload is long-running and you have an endpoint to receive callbacks, register a webhook and let sference notify you when each response finishes.
Submit once with the Batch API
When the request set is fixed, the Batch API is simpler and cheaper than fanning out yourself: you upload one JSONL file, create a batch, and collect results when it completes. sference owns the queue and schedules the rows.
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY)
# 1. Write the JSONL — one request per line
with open("reviews.jsonl", "w") as f:
for i, review in enumerate(REVIEWS):
f.write(json.dumps({
"custom_id": f"review-{i}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "zai-org/GLM-5.3",
"messages": [{"role": "user", "content": f"Reply with one word, positive or negative:\n\n{review}"}],
"max_tokens": 16,
},
}) + "\n")
# 2. Upload and create the batch
uploaded = client.files.create(file=open("reviews.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=uploaded.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
# 3. Poll, then download
while True:
b = client.batches.retrieve(batch.id)
if b.status in ("completed", "failed", "expired", "cancelled"):
break
time.sleep(10)
content = client.files.content(b.output_file_id).text
for line in content.splitlines():
row = json.loads(line)
print(row["custom_id"], row["response"]["body"]["choices"][0]["message"]["content"])Full details — row body shapes, create-time validation, custom_id correlation, and the results endpoint — are in Batch inference.
Choosing between them
- Fixed dataset, no per-item interaction → Batch API. Less code, sference schedules it, and you get one artifact at the end.
- Dynamic workload — you are generating requests as results arrive, or a user is waiting on some of them → background responses with a concurrency bound.
- Need per-item cancellation or retry → background responses. A batch is one unit.
- Tens of thousands of rows, results needed later → Batch API. Uploading one file beats managing 50k connections.
The two compose: use the Batch API for the bulk backfill and background responses for the interactive slice, on the same account and the same 24h window.