# Batch inference

> Enqueue thousands of rows on the 24h window: POST /v1/batches, poll status, download JSONL results.

Source: https://sference.com/docs/guides/batches

**Batch** is the big hammer on sference. Reach for it when volume, not latency, is the constraint: send an array of inference rows and let the scheduler place work on spot-friendly European GPUs, completing within the **24h** completion window (`"24h"` is the only supported value). For user-facing inference, use **realtime** sync endpoints (`/v1/chat/completions`, [`/v1/messages`](https://sference.com/docs/anthropic), blocking `/v1/responses`); for discounted sync work that can tolerate queuing, use [`service_tier: "flex"`](https://sference.com/docs/guides/responses#processing-modes) on `/v1/chat/completions` or `/v1/responses`.

> **OpenAPI is canonical**
>
> Use **API Reference → Inference → Batch API** for field-level detail on `POST /v1/batches`, cancellation, and the results download routes.

## Typical control loop

1. **`POST /v1/batches`**: send an inline `requests[]` array (each row has optional `custom_id` and a `body` object).
2. **`GET /v1/batches/{batch_id}`**: poll `status`, token totals, and `request_count` while work drains.
3. **`GET /v1/batches/{batch_id}/results`** or **`GET /v1/batches/{batch_id}/results.jsonl`**: fetch per-row outcomes once the batch is terminal (`completed`, `failed`, or `cancelled`).

Retries should be **idempotent**: set a stable **`custom_id`** per row when you need correlation across replays.

## One model per batch

Every row in `requests[]` must use the **same** `body.model`. Mixing models in one batch returns **HTTP 400** before anything is enqueued:

```text
requests[1] (custom_id="row-b"): all batch rows must use the same model; expected "Qwen/Qwen3.6-35B-A3B", got "other-model"
```

The model must also be **available in your platform catalog** (`platform_status=available`). Unknown or unavailable ids fail at create with `Model "…" is not available for inference`. Split multi-model workloads into separate batch jobs (or use [Responses & streams](https://sference.com/docs/guides/responses) with per-request models).

## Request shape

```json
{
  "window": "24h",
  "requests": [
    {
      "custom_id": "row-a",
      "body": {
        "model": "Qwen/Qwen3.6-35B-A3B",
        "messages": [{ "role": "user", "content": "Hello" }]
      }
    },
    {
      "custom_id": "row-b",
      "body": {
        "model": "Qwen/Qwen3.6-35B-A3B",
        "messages": [{ "role": "user", "content": "Summarize this." }]
      }
    }
  ]
}
```

Base URL: **`https://api.sference.com/v1`** (Bearer API key).

## Row `body` shapes (normalize → validate)

At create time the API **normalizes** each row `body` to internal chat-completions format, **validates** it, then persists. Invalid rows return **HTTP 400** with `requests[i]` and optional `custom_id`. Nothing is enqueued, so you never get late worker failures like `body.messages must be a list`.

### Chat completions

```json
{
  "model": "Qwen/Qwen3.6-35B-A3B",
  "messages": [{ "role": "user", "content": "Summarize this." }],
  "temperature": 0.2,
  "max_tokens": 512
}
```

`messages` must be a **non-empty** array. Optional fields match `POST /v1/chat/completions` (`tools`, `tool_choice`, …).

### Responses API (normalized at create)

Use the same fields as `POST /v1/responses`. The API converts `input` → `messages`, `max_output_tokens` → `max_tokens`, and similar mappings before enqueue:

```json
{
  "model": "Qwen/Qwen3.6-35B-A3B",
  "input": [{ "role": "user", "content": "Summarize this." }],
  "instructions": "Reply in one sentence.",
  "max_output_tokens": 512
}
```

String shorthand for `input` is supported. After create, stored rows always contain `messages`; workers never see raw Responses shape.

### Rejected at create

| Case | Example error |
|------|----------------|
| Missing `messages` and `input` | `body must include a non-empty messages list or Responses API input` |
| Empty `messages: []` | `body.messages must be a non-empty list` |
| Invalid Responses payload | Field-level validation on `input`, etc. |
| `background: true` in row body | `background is not supported in batch request bodies` |
| Unknown or unavailable model | `requests[i]`: `Model "…" is not available for inference` |
| Mixed models across rows | `requests[i]`: `all batch rows must use the same model; expected "…", got "…"` |

> **Do not set background on batch rows**
>
> Batches are already asynchronous. Use `background: true` on **`POST /v1/responses`** for per-request async jobs, not inside batch row bodies.

## JSONL via CLI / SDK

For file-based workflows, use **`sference batch submit`** or **`client.submit_batch(input_file=...)`**. Each JSONL line is one batch row. Use the same model on every line (or pass **`--model`** / `model=` for content-only lines only).

**OpenAI-style envelope**: the SDK sends only `custom_id` + inner `body`; `method` / `url` are ignored:

```jsonl
{"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"…","messages":[{"role":"user","content":"hi"}]}}
{"custom_id":"b","method":"POST","url":"/v1/responses","body":{"model":"…","input":[{"role":"user","content":"hi"}]}}
```

**Content-only**: requires global `model=` on submit:

```jsonl
{"content":"Classify this log line."}
```

See **[CLI](https://sference.com/docs/cli)** and the [OSS CLI README](https://github.com/s-ference/sference/blob/main/cli/README.md) for subcommands and **`sference batch stream`**.

## Not OpenAI’s file-upload Batch API

sference batches differ from [OpenAI Batch](https://platform.openai.com/docs/guides/batch):

- No **`POST /v1/files`** or batch objects referencing uploaded JSONL file IDs.
- Create uses **inline `requests[]`**, not a separate file upload step.
- Result rows use **`result_json` / `error_json`**, not OpenAI’s batch result envelope.

For OpenAI **Responses** workloads at scale, use Responses-shaped **`body`** rows in a batch (above) or **`POST /v1/responses`** with `background: true` per request.

## CLI quick example

```bash
export SFERENCE_API_KEY='sk_...'
sference auth login --api-key "$SFERENCE_API_KEY"
sference batch submit --input-file ./workload.jsonl --model Qwen/Qwen3.6-35B-A3B --window 24h
sference batch wait --batch-id <batch_id> --timeout 86400
sference batch download-results --batch-id <batch_id> --out ./results.jsonl
```

## Python SDK quick example

```python
from sference_sdk import SferenceClient

client = SferenceClient(api_key="sk_...")
batch = client.submit_batch(
    input_file="./workload.jsonl",
    model="Qwen/Qwen3.6-35B-A3B",
    window="24h",
)
done = client.wait_for_completion(batch.id, poll_interval=5.0, timeout=86_400.0)
client.download_results_jsonl(done.id, out="./results.jsonl")
```

Pass an explicit **`timeout`** to `wait_for_completion`; the default is 30 seconds.

## When *not* to use batches

- **Interactive** assistants with typing indicators → realtime [Responses & streams](https://sference.com/docs/guides/responses).
- **Latency-tolerant sync work** (unattended agents, retries OK) → `service_tier: "flex"` on the sync endpoints is simpler than a batch job; see [Processing modes](https://sference.com/docs/guides/responses#processing-modes).

## Cross-links

- [Quickstart](https://sference.com/docs/quickstart): Authenticate and hit `/v1` for the first time.
- [Models](https://sference.com/docs/models): Match model tier to workload cost.
- [CLI](https://sference.com/docs/cli): JSONL submit, wait, and download.
- [Python SDK](https://sference.com/docs/sdk): submit_batch and agent prompts.
- [API Reference](https://sference.com/docs/api-reference): Batch routes with live schemas.