# Requests at scale

> Move tens of thousands of requests through sference without serializing them one by one.

Source: https://sference.com/docs/guides/requests-at-scale

A loop that fires requests sequentially stalls as soon as the workload grows past a few hundred calls. sference gives you two ways to clear large volumes: **fan out concurrent background responses**, or hand a single bundle to the **Batch API** and let sference work through it.

## Which path fits your workload

- [Concurrent background responses](#fan-out-with-background-responses): You want results as each one finishes, need per-request control, or care about latency on individual items. You manage concurrency.
- [Batch API](#submit-once-with-the-batch-api): You have a fixed set of requests and would rather submit once and collect later. sference owns the queue.

| | Background responses | Batch API |
| --- | --- | --- |
| Unit of work | one HTTP request each | one JSONL file |
| Results | poll each response id, or [webhook](https://sference.com/docs/guides/webhooks) | poll the batch, download JSONL |
| Concurrency | you bound it | sference schedules it |
| Per-item control | full — cancel, retry, inspect individually | limited to the batch |
| Best for | dynamic workloads, agent fan-out, streaming results | fixed datasets, evals, offline jobs |
| Window | 24h batch | 24h batch |

The running example below is sentiment scoring for a backlog of product reviews.

## Fan out with background responses

Submit each request with `background: true` and keep many in flight. For workloads of roughly 1,000 requests and up, four practices keep it stable:

- **Use `AsyncOpenAI` with `DefaultAioHttpClient()`.** Under high concurrency the aiohttp backend outperforms the default `httpx` transport.
- **Bound concurrency with an `asyncio.Semaphore`.** A fixed limit (200 is a sound starting point) caps simultaneous connections so you never exhaust them.
- **Submit all requests first, then poll in a second pass.** Separating submission from polling keeps both phases simple and avoids interleaving.
- **Set `max_queue_wait_s`** so a congested moment fails fast instead of holding connections open — see [Max queue wait](https://sference.com/docs/guides/max-queue-wait).

```bash
pip install tqdm 'openai[aiohttp]'
```

```python
import asyncio
from tqdm import tqdm
from openai import AsyncOpenAI, DefaultAioHttpClient

CONCURRENCY = 200

async def score(client, review: str) -> str:
    response = await client.responses.create(
        model="zai-org/GLM-5.3",
        input=f"Reply with one word, positive or negative:\n\n{review}",
        background=True,
        max_output_tokens=16,
        max_queue_wait_s=30,
    )
    return response.id

async def main(reviews: list[str]):
    async with AsyncOpenAI(
        base_url="https://api.sference.com/v1",
        api_key=SFERENCE_KEY,
        http_client=DefaultAioHttpClient(),
    ) as client:
        sem = asyncio.Semaphore(CONCURRENCY)

        async def submit(review):
            async with sem:
                return await score(client, review)

        # Pass 1: submit everything, collect ids
        ids = []
        for coro in tqdm(asyncio.as_completed([submit(r) for r in reviews]), total=len(reviews)):
            ids.append(await coro)

        # Pass 2: poll to completion
        results = {}
        pending = set(ids)
        while pending:
            for rid in list(pending):
                r = await client.responses.retrieve(rid)
                if r.status in ("completed", "failed", "incomplete"):
                    results[rid] = r
                    pending.discard(rid)
            if pending:
                await asyncio.sleep(1)

        return results

asyncio.run(main(REVIEWS))
```

> **Collect on the finish reason, not on status alone**
>
> On a reasoning model, `max_output_tokens` covers hidden reasoning as well as the answer. A response can settle `incomplete` with a perfectly good answer attached. Check the incomplete detail and use the output if it is there, rather than discarding it.

### Prefer webhooks to polling

Polling every response id in a loop is the simplest thing that works, but it is wasteful at scale. If your workload is long-running and you have an endpoint to receive callbacks, register a [webhook](https://sference.com/docs/guides/webhooks) and let sference notify you when each response finishes.

## Submit once with the Batch API

When the request set is fixed, the [Batch API](https://sference.com/docs/guides/batches) is simpler and cheaper than fanning out yourself: you upload one JSONL file, create a batch, and collect results when it completes. sference owns the queue and schedules the rows.

```python
import json
from openai import OpenAI

client = OpenAI(base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY)

# 1. Write the JSONL — one request per line
with open("reviews.jsonl", "w") as f:
    for i, review in enumerate(REVIEWS):
        f.write(json.dumps({
            "custom_id": f"review-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "zai-org/GLM-5.3",
                "messages": [{"role": "user", "content": f"Reply with one word, positive or negative:\n\n{review}"}],
                "max_tokens": 16,
            },
        }) + "\n")

# 2. Upload and create the batch
uploaded = client.files.create(file=open("reviews.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=uploaded.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

# 3. Poll, then download
while True:
    b = client.batches.retrieve(batch.id)
    if b.status in ("completed", "failed", "expired", "cancelled"):
        break
    time.sleep(10)

content = client.files.content(b.output_file_id).text
for line in content.splitlines():
    row = json.loads(line)
    print(row["custom_id"], row["response"]["body"]["choices"][0]["message"]["content"])
```

Full details — row body shapes, create-time validation, `custom_id` correlation, and the results endpoint — are in [Batch inference](https://sference.com/docs/guides/batches).

## Choosing between them

- **Fixed dataset, no per-item interaction** → Batch API. Less code, sference schedules it, and you get one artifact at the end.
- **Dynamic workload** — you are generating requests as results arrive, or a user is waiting on some of them → background responses with a concurrency bound.
- **Need per-item cancellation or retry** → background responses. A batch is one unit.
- **Tens of thousands of rows, results needed later** → Batch API. Uploading one file beats managing 50k connections.

The two compose: use the Batch API for the bulk backfill and background responses for the interactive slice, on the same account and the same 24h window.

- [Batch inference](https://sference.com/docs/guides/batches): JSONL jobs, body shapes, and create-time validation.
- [Webhooks](https://sference.com/docs/guides/webhooks): Stop polling: get notified when work finishes.
- [Max queue wait](https://sference.com/docs/guides/max-queue-wait): Fail fast under congestion instead of holding connections.