# sference: full documentation > Every documentation page on sference.com concatenated as Markdown. Each page is also available on its own by appending `.md` to its URL (e.g. https://sference.com/docs/api/chat-completions.md). Shorter index: https://sference.com/llms.txt ## Contents - [Overview](https://sference.com/docs.md) - [Quickstart](https://sference.com/docs/quickstart.md) - [Models](https://sference.com/docs/models.md) - [Pricing](https://sference.com/docs/pricing.md) - [Usage & analytics](https://sference.com/docs/usage.md) - [Sference Switch](https://sference.com/docs/switch.md) - [Python SDK](https://sference.com/docs/sdk.md) - [CLI](https://sference.com/docs/cli.md) - [LLM & agent access](https://sference.com/docs/llm-access.md) - [Pricing for agents](https://sference.com/docs/pricing-for-agents.md) - [API compatibility](https://sference.com/docs/compatibility.md) - [Support matrix](https://sference.com/docs/support.md) - [Custom Models](https://sference.com/docs/custom-models.md) - [Deployments](https://sference.com/docs/deployments.md) - [Build with agent frameworks](https://sference.com/docs/agents.md) - [LangChain](https://sference.com/docs/agents/langchain.md) - [OpenAI Agents SDK](https://sference.com/docs/agents/openai-agents-sdk.md) - [Anthropic Agent SDK](https://sference.com/docs/agents/anthropic-agent-sdk.md) - [API surfaces](https://sference.com/docs/api.md) - [OpenAI Chat API](https://sference.com/docs/api/chat-completions.md) - [OpenAI Responses API](https://sference.com/docs/api/responses.md) - [Anthropic Messages API](https://sference.com/docs/api/messages.md) - [Batch inference](https://sference.com/docs/guides/batches.md) - [Streams](https://sference.com/docs/guides/streams.md) - [Prompt caching](https://sference.com/docs/guides/prompt-caching.md) - [Images and vision](https://sference.com/docs/guides/vision.md) - [Max queue wait](https://sference.com/docs/guides/max-queue-wait.md) - [Webhooks](https://sference.com/docs/guides/webhooks.md) - [Decisions](https://sference.com/docs/guides/decisions.md) - [Requests at scale](https://sference.com/docs/guides/requests-at-scale.md) - [Structured outputs](https://sference.com/docs/guides/structured-outputs.md) - [SferenceCode](https://sference.com/docs/sference-code.md) - [Claude Code](https://sference.com/docs/sference-code/claude-code.md) - [Pi](https://sference.com/docs/sference-code/pi.md) - [opencode](https://sference.com/docs/sference-code/opencode.md) - [Codex CLI](https://sference.com/docs/sference-code/codex.md) - [Cursor](https://sference.com/docs/sference-code/cursor.md) - [API Reference](https://sference.com/docs/api-reference.md) The HTTP API reference is generated from the OpenAPI specification, which is machine-readable at https://sference.com/openapi.json and browsable at https://sference.com/docs/api-reference. Every reference page also has a Markdown twin: https://sference.com/docs/api-reference.md lists all operations and links to each one. --- # sference > Inference stack for open models. One OpenAI-compatible API serving both realtime AI and long-running AI agents, with realtime, flex, and batch processing modes for open-weight models, fine-tunes, and distilled models. Independently measured as the fastest provider on the models we serve through the Requesty gateway. Full logging, auditing, and compliance built in. ## What is sference? sference is a managed inference platform for open models. We run a scheduler and hardware-agnostic inference runtime built from the ground up, exposed as one OpenAI-compatible API. It serves both shapes of production AI from the same fleet: realtime AI, where a user is waiting on the response, and long-running AI agents, where work spans many model calls and nobody is waiting on any single one. Realtime, flex, and batch are scheduling priorities on one system, not separate deployments. The platform is high-performance and efficient at the same time: the design that keeps latency low is the same one that keeps GPU utilization high. Companies use it as one platform for all their inference needs, whether catalog open-weight models or their own custom weights, with request-level logging, auditing, and compliance built in, and without building and operating an inference stack themselves. ## Who is it for? Series A+ companies and enterprises running AI in production who want off the closed-API treadmill for cost, control, or compliance reasons. Teams migrating workloads from OpenAI/Anthropic onto open models, teams with fine-tuned or distilled models that need production-grade serving, and companies in regulated industries whose customers ask where their data goes. ## Key capabilities - **Any open model, including yours**: Open-weight frontier models: Kimi K3 (vision), GLM 5.3, DeepSeek V4.1 Flash (vision), GLM 5.3 Flash (vision), DeepSeek V4 Flash (0731), and Clef 27B (realtime decisions, vision) live today; DeepSeek V4 Pro, Gemma 4 31B, NVIDIA Nemotron 3 Super, OpenAI GPT-OSS 120B, and Qwen3.8 coming soon. Or upload your own weights: fine-tunes and distilled models run on the same API, infrastructure, and audit trail as catalog models. vLLM/SGLang models run out of the box; we stand up custom runtimes for models that don't. Custom model hosting is arranged per customer as part of an enterprise agreement. - **Realtime, flex, and batch on the same API**: Sync endpoints (`/v1/chat/completions`, `/v1/messages`, blocking `/v1/responses`) for interactive inference; `service_tier: "flex"` for discounted latency-tolerant sync work; and a **24h** completion window for batches, background responses, and offline pipelines. All three share the same models, endpoints, and fleet. OpenAI's Flex processing is supported: requests with `service_tier: "flex"` get discounted tokens at lower scheduling priority (best-effort, 408 + retry on timeout). OpenAI-compatible, so most clients switch with a base-URL change. - **Auto model routing (beta)**: `sference/auto` picks a model per session from the first user message — low `deepseek-ai/DeepSeek-V4.1-Flash`, medium `zai-org/GLM-5.3`, high `moonshotai/Kimi-K3` — and keeps it for the session so the prompt cache stays warm. The response `model` is the routed model, and `x-sference-routed-model` / `x-sference-route-tier` headers report the decision. Off by default, enabled per organization by an org admin; `/v1/chat/completions`, `/v1/messages`, and `/v1/responses` only (batches reject it). Docs: https://sference.com/docs/models#auto-model-routing - **Anthropic-compatible too**: `POST /v1/messages` speaks the Anthropic Messages API, so the `anthropic` Python/TypeScript SDKs, Claude Code, and any Messages client run on open models by changing the base URL to `https://api.sference.com` and the model to a sference catalog id. Streaming SSE, tool use, extended thinking, and vision are supported; `x-api-key` and `Authorization: Bearer` both authenticate. Realtime only; flex, background, and batch live on the OpenAI-shaped endpoints. Docs: https://sference.com/docs/api/messages. Claude Code setup (one command: `sference launch claude`, with hybrid routing so Claude models still pass through to Anthropic): https://sference.com/docs/sference-code/claude-code - **Serverless to dedicated**: Start on shared endpoints with per-token pricing and no minimums. Move hot paths to dedicated deployments (reserved GPU capacity, private endpoints, predictable latency), billed per GPU-hour on monthly commitments. Enterprise agreements add volume pricing, custom SLAs, and procurement-friendly terms. - **Fastest measured route**: Independent live data (Requesty Model Discovery, gateway-wide provider comparison, mid-August 2026): sference serves `kimi-k3` at 426 ms average latency / 76 tok/s, `deepseek-v4-flash` at 861 ms / 157 tok/s, and `glm-5.2` at 3.1 s / 211 tok/s. That is the highest throughput of any provider on all three, and the lowest average latency on `kimi-k3` and `deepseek-v4-flash`. - **High-performance and efficient, from one design**: A scheduler with priority queues and backpressure at the core. Realtime, flex, and batch windows are scheduling priorities on one system, not separate stacks, so interactive traffic never queues behind a backfill. Requests are recorded and dispatched transactionally and routed over per-model queues (the queue is the router, so no load balancer guesses at GPU state). Workers drive vLLM and SGLang as in-process libraries rather than servers behind another network hop, giving faster first tokens and higher throughput, with continuous batching, prefix caching, and bounded in-flight slots, so realtime traffic packs onto the same GPUs as batch backfill. Low latency and high utilization come from the same scheduler; utilization around the clock is where the per-token prices come from. - **Forward-deployed onboarding**: Our engineers benchmark your real prompts across candidate open models side by side against your current provider, then work with your team through prompt migration and integration until the workload runs in production. - **Pinned model versions**: Pin an exact version: every release has its own versioned id, so the version you validated is the version that serves. The serving version is recorded on every request, and a newer version can be validated by rerunning the same evals before switching. - **Pipeline-friendly SDK**: Python SDK for batch and background inference from the orchestration you already run: submit batches, fan out background responses, collect results. Worked Prefect example in the docs. - **Request-level logging and full audit trail**: Every request logged with model version, infrastructure region, timestamps, and data residency. The same logging and audit trail applies to catalog and custom models, realtime and batch alike. Configurable retention. Exportable compliance reports. ## Pricing - **Serverless**: pay per token, no minimum spend. Serverless rates (USD per 1M tokens, realtime priority; confirmed per model at account provisioning; the site shows USD with a EUR toggle): Kimi K3 (vision) $3.00 / $15.00 / $0.45 cached; GLM 5.3 $1.20 / $4.20 / $0.26 cached; DeepSeek V4.1 Flash (vision) $0.50 / $1.50 / $0.05 cached; GLM 5.3 Flash (vision) $0.20 / $0.60 / $0.07 cached; DeepSeek V4 Flash (0731) $0.28 / $0.56 / $0.07 cached; Clef 27B (realtime decisions, vision) $0.24 / — / —. Coming soon: DeepSeek V4 Pro, Gemma 4 31B, NVIDIA Nemotron 3 Super, OpenAI GPT-OSS 120B, and Qwen3.8. Live rates: https://api.sference.com/v1/models. Batch and flex windows are discounted against the realtime rate. Custom model hosting and pricing are agreed per customer as part of an enterprise agreement. - **Dedicated**: reserved GPU capacity with private endpoints, billed per GPU-hour on monthly commitments. Contact us. - **Enterprise**: annual agreements, committed-spend discounts, volume pricing, custom SLAs, forward-deployed onboarding included. Contact us. - **Why per-token beats self-managed**: our scheduler keeps GPU utilization high around the clock; on self-managed instances or on-prem clusters, idle capacity is paid for whether it serves tokens or not. ## Compliance and regulation - GDPR compliant - DORA ready (in enforcement since January 2025) - EU AI Act deployer obligations ready (effective August 2026) - DPA included; public copy at https://sference.com/legal/dpa; sub-processor list at https://sference.com/legal/subprocessors - Configurable data retention policies - Exportable compliance reports for security reviews and audits - Zero training on customer data, contractually and technically ## Workloads Production workloads teams run on sference. Overview at https://sference.com/#use-cases. - **Coding and internal assistants**: Copilots, code review, and coding agents on open models, alongside the internal chat surfaces teams already run, such as Open WebUI. Streaming, tool calling, and structured output; Kimi K3 (vision), GLM 5.3, DeepSeek V4.1 Flash (vision), GLM 5.3 Flash (vision), DeepSeek V4 Flash (0731), and Clef 27B (realtime decisions, vision) in the catalog, your fine-tunes on the same API. - **Features powered by agents**: Product features built on interactive and background agents. The turn a user waits on runs realtime; the steps behind it run on flex or batch windows, under one API and one audit trail, with queues and backpressure that keep long jobs from falling over and pinned versions so the checkpoint behind an agent stays explicit. ## Verticals - **FinTech**: Batch KYC extraction, transaction classification, statement processing - **LegalTech**: Contract corpus analysis, document review, embedding generation for legal RAG - **HealthTech**: Medical record digitization, prescription extraction, clinical data processing - **InsureTech**: Claims processing, policy document analysis, underwriting data extraction - **AI/ML Teams**: Model serving for fine-tunes and distillations, evals, synthetic data generation ## Processing modes - **realtime**: sync `/v1/chat/completions`, `/v1/messages`, or blocking `/v1/responses` when a user is waiting - **flex**: `/v1/chat/completions` or `/v1/responses` with `service_tier: "flex"`: discounted tokens at lower scheduling priority; may queue longer (up to 30 minutes), fails with 408 on timeout (OpenAI SDKs retry automatically); enabled per account. Not available on `/v1/messages`, which is realtime only - **batch (24h)**: background responses (`background: true`), streams, and batches on the 24h completion window, which is the only supported batch window value ## API Base URL: https://api.sference.com/v1/, with one API key (`sk_...`) for every surface. - **OpenAI-compatible**: `/v1/chat/completions`, `/v1/responses` (adds `background: true` and the 24h batch window), `/v1/embeddings`, `/v1/batches`, `/v1/streams`, `/v1/models` - **Anthropic-compatible**: `/v1/messages` (Messages API shape; realtime only) - **Decisions**: `/v1/decisions` (structured questions and calibrated probabilities; realtime only) - **Control plane**: `/control/v1/...` for teams, API keys, catalog, deployments, activity, usage, billing - Machine-readable OpenAPI specification: https://sference.com/openapi.json ## Model cards One card per model in the live catalog: pricing, context and capabilities, how the model fits the portfolio, a quickstart, and curated benchmarks, serving details, and FAQ where available. Generated from the catalog, so they never drift. Index: https://sference.com/models (also as Markdown: https://sference.com/models.md — every card too, by appending `.md`). - [Kimi K3](https://sference.com/models/moonshotai/Kimi-K3) - [GLM 5.3](https://sference.com/models/zai-org/GLM-5.3) - [DeepSeek V4.1 Flash](https://sference.com/models/deepseek-ai/DeepSeek-V4.1-Flash) - [GLM 5.3 Flash](https://sference.com/models/zai-org/GLM-5.3-Flash) - [DeepSeek V4 Flash (0731)](https://sference.com/models/deepseek-ai/DeepSeek-V4-Flash-0731) - [Clef 27B](https://sference.com/models/Cloudflare/clef) ## Documentation Full docs as a single Markdown file: https://sference.com/llms-full.txt Any docs page as Markdown: append `.md` to its URL — guide pages and every API-reference page (e.g. https://sference.com/docs/api/chat-completions.md or https://sference.com/docs/api-reference/inference-api/V1ChatCompletionsCreateChatCompletion.md) Automatic: send `Accept: text/markdown` on any /docs URL and receive Markdown without the suffix Blog posts as Markdown: append `.md` to any /blog post URL (e.g. https://sference.com/blog/sference-is-live-on-opper.md) Legal pages as Markdown: https://sference.com/legal/aup.md, https://sference.com/legal/dpa.md, https://sference.com/legal/subprocessors.md, https://sference.com/privacy.md, https://sference.com/terms.md - [Overview](https://sference.com/docs): hub, three processing modes, where each client lives - [Quickstart](https://sference.com/docs/quickstart): API key, CLI install, first request, curl fallback - [Models](https://sference.com/docs/models): choosing a tier, pinning versions, aliases, auto model routing, catalog and BYOM entry points - [Pricing](https://sference.com/docs/pricing): serverless rates per model, the three windows, cached input - [OpenAI Chat API](https://sference.com/docs/api/chat-completions): `POST /v1/chat/completions`: the OpenAI-shaped path, streaming, tools, structured output - [OpenAI Responses API](https://sference.com/docs/api/responses): `POST /v1/responses`: stateful responses, events, background mode, the realtime/flex/batch mode table - [Anthropic Messages API](https://sference.com/docs/api/messages): `POST /v1/messages`: auth, supported fields and content blocks, SSE event sequence, tool use, extended thinking, images, errors, and the exact differences from api.anthropic.com - [Custom Models](https://sference.com/docs/custom-models): bring your own Hugging Face weights, so fine-tunes and distilled models run on the same API - [Deployments](https://sference.com/docs/deployments): dedicated reserved capacity for a base or custom model - [Batch inference](https://sference.com/docs/guides/batches): one model per batch, catalog validation at create, chat or Responses row bodies, JSONL results, 24h window - [Streams](https://sference.com/docs/guides/streams): group related requests into one window with a cost rollup and a resumable cursor - [Prompt caching](https://sference.com/docs/guides/prompt-caching): keep the prefix stable and pin the worker to cut cached input cost - [Images and vision](https://sference.com/docs/guides/vision): which models accept images, the request shapes, and how they bill - [Max queue wait](https://sference.com/docs/guides/max-queue-wait): bound the wait and fail over instead of hanging - [Webhooks](https://sference.com/docs/guides/webhooks): get a callback when a background response finishes instead of polling (beta) - [Decisions](https://sference.com/docs/guides/decisions): `POST /v1/decisions`: choice, score, and yes/no questions with calibrated probabilities in one realtime call - [Usage and analytics](https://sference.com/docs/usage): spend, tokens, and per-request activity grouped by model, key, member, and department - [Build with agent frameworks](https://sference.com/docs/agents): LangChain, the OpenAI Agents SDK, and the Anthropic Agent SDK, plus structured output and scale patterns - [Structured outputs](https://sference.com/docs/guides/structured-outputs): constrain the response to a JSON schema - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): fan out concurrent background responses, or submit one batch - [Claude Code](https://sference.com/docs/sference-code/claude-code): `sference launch claude` proxy mode with hybrid routing, manual `ANTHROPIC_BASE_URL` config, model recommendations, known limits, troubleshooting - [SferenceCode](https://sference.com/docs/sference-code): run Claude Code, Pi, opencode, Codex, or Cursor on open models, one command each - [Python SDK](https://sference.com/docs/sdk): `pip install sference-sdk`, default `create_response(background=True)` + `wait_for_response`, `submit_batch` for JSONL jobs, Prefect example - [CLI](https://sference.com/docs/cli): `curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh`: browser sign-in (`sference auth login`, OAuth device flow; `--api-key` for CI), batch, responses, stream, and `sference launch claude|codex|opencode|pi` to run your coding agent on sference models for one session - [Sference Switch](https://sference.com/docs/switch): macOS app (`curl -fsSL https://get.sference.com | sh`) with one on/off switch that routes every Claude Code and Codex session on the machine to sference models; Claude models still pass through to Anthropic - [LLM and agent access](https://sference.com/docs/llm-access): Markdown views, llms.txt, and the OpenAPI spec for coding agents - [Pricing for agents](https://sference.com/docs/pricing-for-agents): model by window rate table for agent workloads - [API compatibility](https://sference.com/docs/compatibility): what each surface honors and what it ignores - [Support matrix](https://sference.com/docs/support): one-page grid of every endpoint, field, and window - [API Reference](https://sference.com/docs/api-reference): every path and schema generated from the OpenAPI specification **Coding agents**: copy OSS **PROMPT.txt** (`https://raw.githubusercontent.com/s-ference/sference/main/PROMPT.txt`) for vibe coding; optional full **SKILL.md** for Cursor skills. Open-source clients: https://github.com/s-ference/sference ## Contact - Website: https://sference.com - Documentation: https://sference.com/docs - Use cases: https://sference.com/#use-cases - Pricing: https://sference.com/#pricing - FAQ: https://sference.com/#faq - Status: https://status.sference.com - Changelog: https://sference.com/changelog - Email: hello@sference.com - LinkedIn: https://www.linkedin.com/company/sference/ --- # Overview > Inference for realtime AI and long-running AI agents: an OpenAI-compatible API with realtime, flex, and batch windows on one fleet. Quickstart, models, pricing, and full HTTP reference. Source: https://sference.com/docs **sference** is an inference stack for open models, exposed as one **OpenAI-compatible** HTTP API. It serves both shapes of production AI from the same fleet: **realtime AI**, where a user is waiting on the response, and **long-running AI agents**, where work spans many model calls and nobody is waiting on any single one. Three processing modes carry that range, and you pick per request: | Mode | Window | For | | --- | --- | --- | | **realtime** | immediate | Interactive requests, the turn a user is waiting on | | **flex** | up to 30 minutes | Latency-tolerant sync work: background steps, enrichment, summarization | | **batch** | up to 24 hours | Large jobs, evals, backfills, offline pipelines | All three run on the same endpoints, the same models, and the same GPU fleet — a scheduling priority per request rather than a separate deployment per workload. Pin model versions and keep a compliance-grade trail, without replatforming how your team already calls models. [Quickstart](https://sference.com/docs/quickstart) · [Models](https://sference.com/docs/models) · [Pricing](https://sference.com/docs/pricing) · [API Reference](https://sference.com/docs/api-reference) ## Start here - [Quickstart](https://sference.com/docs/quickstart): API key, CLI install, and your first response, curl optional. - [Models](https://sference.com/docs/models): Catalog ids, capabilities, context windows, and how to pin versions. - [Pricing](https://sference.com/docs/pricing): Per-token rates, the three windows, cached input, and what counts as a token. - [Support matrix](https://sference.com/docs/support): One-page grid: every endpoint, field, and window. ## Developer quickstart Export an API key and make a realtime request. It is OpenAI-shaped, so an existing client needs only a new base URL and model id: ```bash export SFERENCE_API_KEY="sk_..." curl https://api.sference.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -d '{"model":"deepseek-ai/DeepSeek-V4-Flash-0731","messages":[{"role":"user","content":"Say hello in one sentence."}]}' ``` Prefer the CLI? Install it and authenticate: ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh sference auth login # opens the console to approve this device ``` Step-by-step: **[Quickstart](https://sference.com/docs/quickstart)**. Prefer Python? See **[Python SDK](https://sference.com/docs/sdk)**. ## Build an agent Agent loops are the workload the platform is shaped for: the turn a user waits on runs realtime, the steps behind it run flex or batch, and one bill covers both. - [Build with agent frameworks](https://sference.com/docs/agents): LangChain, the OpenAI Agents SDK, and the Anthropic Agent SDK, plus structured output and scale patterns. - [SferenceCode](https://sference.com/docs/sference-code): Run Claude Code, Pi, opencode, Codex, or Cursor on open models, one command each. - [Prompt caching](https://sference.com/docs/guides/prompt-caching): The largest lever on an agent's bill: keep the prefix stable, pin the worker. - [Max queue wait](https://sference.com/docs/guides/max-queue-wait): Bound the wait and fail over instead of hanging. ## Choose a workload - [OpenAI Responses API](https://sference.com/docs/api/responses): Realtime and background responses, events, and `/v1/streams` for incremental output. - [Batch API](https://sference.com/docs/guides/batches): Enqueue large jobs on the 24h window: chat or Responses row bodies, poll status, download JSONL. - [Streams](https://sference.com/docs/guides/streams): Group related requests: one window, one cost rollup, a resumable cursor. - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): Fan out concurrent background responses, or submit one batch. - [Webhooks](https://sference.com/docs/guides/webhooks): Get a callback when a background response finishes instead of polling. - [Structured outputs](https://sference.com/docs/guides/structured-outputs): Constrain the response to a JSON schema. - [Images and vision](https://sference.com/docs/guides/vision): Which models accept images, the request shapes, and how they bill. - [Max queue wait](https://sference.com/docs/guides/max-queue-wait): Fail fast under congestion; adopt sference as a preferred provider. ## Operate it - [Usage & analytics](https://sference.com/docs/usage): Spend, tokens, and per-request activity, grouped by model, key, member, and department. - [API compatibility](https://sference.com/docs/compatibility): What each surface honors, and what it ignores. - [LLM & agent access](https://sference.com/docs/llm-access): Markdown views, llms.txt, and the OpenAPI spec for coding agents. - [HTTP reference](https://sference.com/docs/api-reference): Every path and schema generated from our OpenAPI spec, including control plane routes. ## Bring your existing client - [Anthropic Messages API](https://sference.com/docs/api/messages): POST /v1/messages: point the anthropic SDK at sference and run open models. - [OpenAI Chat API](https://sference.com/docs/api/chat-completions): POST /v1/chat/completions: the OpenAI-shaped path most clients already speak. - [OpenAI Responses API](https://sference.com/docs/api/responses): POST /v1/responses: stateful responses, events, and background mode. - [Claude Code](https://sference.com/docs/sference-code/claude-code): One command to run Claude Code on open models, with hybrid routing. OpenAI-shaped clients need only a base-URL change: `/v1/chat/completions`, `/v1/responses`, and `/v1/embeddings` are OpenAI-compatible. For classification and routing, [`/v1/decisions`](https://sference.com/docs/guides/decisions) returns probabilities for structured questions instead of generated text. ## Product & compliance For positioning, regulation, and workloads, use the [marketing site](https://sference.com/) and [use cases](https://sference.com/#use-cases). --- # Quickstart > Create an API key, install the CLI, and run your first inference request. Source: https://sference.com/docs/quickstart The **Inference API** is an OpenAI-shaped surface, with **realtime**, **flex**, and **batch** processing modes for anything from an interactive request a user is waiting on to an overnight batch job. The fastest path is the **`sference` CLI**; use the [Python SDK](https://sference.com/docs/sdk) or raw HTTP when you need them in your stack. > **Control plane vs inference** > > - **Inference** (`https://api.sference.com/v1/...`): models, batches, responses, streams. > - **Control plane** (`/control/v1/...` in **API Reference**): orgs, API keys, catalog, billing. > > This page focuses on **inference**. Keys are minted in the [sference console](https://app.sference.com). ## Create and export an API key 1. Sign in to sference and create an **API key** for your workspace. 2. Store it somewhere safe (password manager, Doppler, Vault, …). Never commit keys to git. 3. Export it for local development: ```bash export SFERENCE_API_KEY="sk_..." ``` On Windows (PowerShell): ```powershell setx SFERENCE_API_KEY "sk_..." ``` The CLI and Python SDK read **`SFERENCE_API_KEY`** from the environment. ## Install the CLI ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh ``` Other options: `uv tool install sference-cli`, `pip install sference-cli`, or `pipx install sference-cli`. See **[CLI](https://sference.com/docs/cli)** for details. ## Run your first request Start with a **realtime** request, a synchronous call that returns when inference finishes. It is OpenAI-shaped, so if you already call `/v1/chat/completions` somewhere, only the base URL and the model id change: ```bash curl https://api.sference.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ { "role": "user", "content": "Write a one-sentence greeting for an inference platform." } ] }' ``` Add `"stream": true` to receive tokens as they are produced rather than waiting for the full completion. ### The same thing from the CLI Sign the CLI in (if you have not already). `sference auth login` opens the console to approve this device; in scripts, use `--api-key` with the key you created above: ```bash sference auth login # or, non-interactive: sference auth login --api-key "$SFERENCE_API_KEY" ``` ```bash sference responses create \ --model "deepseek-ai/DeepSeek-V4-Flash-0731" \ --content "Write a one-sentence greeting for an inference platform." \ --wait ``` `--wait` returns the finished result instead of a response id. Add `--json` for machine-readable output. For batches, streams, and JSONL jobs, see **[CLI](https://sference.com/docs/cli)** and the [OSS README](https://github.com/s-ference/sference). ## Realtime, flex, or batch? The example above runs on the **realtime** mode, a synchronous request that returns when inference finishes. All three modes share the same endpoint, models, and fleet: | Mode | How | When | |------|-----|------| | **Realtime** | Sync `/v1/chat/completions`, [`/v1/messages`](https://sference.com/docs/api/messages), or blocking `/v1/responses` | A user is waiting on the response | | **Flex** | `/v1/chat/completions` or `/v1/responses` with `service_tier: "flex"` (not `/v1/messages`) | Latency-tolerant sync work at a discount; may queue longer, retries on 408 | | **Batch** | `background: true` on `/v1/responses`, streams, batches | Large jobs, evals, and offline pipelines on the 24h completion window | ## Python SDK For application code or [coding agents](https://sference.com/docs/sdk): ```bash pip install sference-sdk ``` For interactive code, call the blocking response/chat methods; for unattended jobs, use `create_response(..., background=True)` with `wait_for_response`; see **[Python SDK](https://sference.com/docs/sdk)**. ## Background work with raw HTTP When nobody is waiting on the result, whether bulk jobs, long generations, or anything off the interactive path, send it to `/v1/responses` with `background: true` and collect it later: ```bash curl https://api.sference.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "input": [ { "role": "user", "content": "Write a one-sentence greeting for an inference platform." } ], "background": true }' ``` You get a **`response`** object with an `id` and `status` (`in_progress`, `completed`, `failed`, …). Poll `GET /v1/responses/{response_id}` or use events; see [OpenAI Responses API](https://sference.com/docs/api/responses). ## Go deeper - [CLI](https://sference.com/docs/cli): Install, auth, batches, streams, and responses subcommands. - [Python SDK](https://sference.com/docs/sdk): Library usage and copy-paste prompts for Cursor and Claude Code. - [Anthropic Messages API](https://sference.com/docs/api/messages): Already have Anthropic-shaped code? Change the base URL and the model id. - [Models](https://sference.com/docs/models): Choosing a tier, pinning versions, and browsing the catalog. - [Batch API](https://sference.com/docs/guides/batches): Thousands of rows on the 24h window, not a single prompt. - [API Reference](https://sference.com/docs/api-reference): OpenAPI-backed pages for every operation. --- # Models > The sference catalog: how to choose a model, pin a version, read capabilities from the API, and bring your own weights. Source: https://sference.com/docs/models Every model on sference is an open-weight checkpoint addressed by an exact catalog id. There is no moving `latest` alias: the id you validate is the id that serves, and the serving version is recorded on every request. That is what makes a model change a deliberate act rather than something you discover in a regression. ## Choosing a model | You need | Start with | | --- | --- | | The cheapest thing that works | `zai-org/GLM-5.3-Flash` | | A general production default | `zai-org/GLM-5.3` | | Long context at low cost | `deepseek-ai/DeepSeek-V4.1-Flash` | | The strongest reasoning in the catalog | `moonshotai/Kimi-K3` | | Image input | `zai-org/GLM-5.3-Flash`, `moonshotai/Kimi-K3`, or `deepseek-ai/DeepSeek-V4.1-Flash` | Rates for each are on the [Pricing](https://sference.com/docs/pricing) page. The pattern that works for most teams: pick the **smallest model that clears your quality bar**, run routine traffic on it, and escalate to a larger model only for the turns that need it. Model choice is a per-request decision — nothing stops you routing easy work to `Flash` and hard work to `Kimi-K3` in the same application. > **Validate on your own prompts** > > Benchmarks are a starting point, not a decision. Run your real prompts through two or three candidates and compare on your own quality bar — the difference between models is usually task-specific, and it is cheap to measure. ## Catalog Every available model, with the capabilities that decide routing. Rates live on the [Pricing](https://sference.com/docs/pricing) page and in the API — see [below](#reading-the-catalog-from-the-api). | Model | Context | Vision | Tools | Thinking | | --- | ---: | :---: | :---: | :---: | | `moonshotai/Kimi-K3` | 1M | ✓ | ✓ | ✓ | | `zai-org/GLM-5.3` | 1M | — | ✓ | ✓ | | `zai-org/GLM-5.2` | 1M | — | ✓ | ✓ | | `deepseek-ai/DeepSeek-V4.1-Flash` | 1M | ✓ | ✓ | ✓ | | `deepseek-ai/DeepSeek-V4-Flash-0731` | 1M | — | ✓ | ✓ | | `deepseek-ai/DeepSeek-V4-Flash` | 1M | — | ✓ | ✓ | | `Cloudflare/clef` | 64K | ✓ | — | — | | `zai-org/GLM-5.3-Flash` | 1M | ✓ | ✓ | ✓ | > **This table is a snapshot** > > `GET https://api.sference.com/v1/models` is authoritative. It reflects your entitlements — including custom models and models agreed in your contract — with capabilities and rates per entry. Read it at runtime rather than hardcoding this table. ### Aliases and retirement Some ids are **aliases**: they resolve to another model and bill at that model's rates. Sending an alias works, and the response reports the model that actually served. | Id | Resolves to | Note | | --- | --- | --- | | `zai-org/GLM-5.2` | `zai-org/GLM-5.3` | Retiring **2026-10-01**; identical rates. Migrate to the `GLM-5.3` id | | `deepseek-ai/DeepSeek-V4-Flash` | `deepseek-ai/DeepSeek-V4-Flash-0731` | Prefer the dated id for new work | > **Aliases are a migration aid, not a target** > > An alias exists so an existing integration keeps working while you migrate. Pin the resolved id in production — an alias can be retired, and the audit trail is cleaner when the id in your code is the id that ran. ## Live catalog and serverless rates The full catalog with exact ids and current serverless rates (USD per 1M tokens, realtime priority). Every model is addressed by its exact id string, and `GET /v1/models` returns the live list your key can reach, including per-model pricing and capabilities. The table below is generated from that endpoint, so it always matches the live catalog; the same catalog, search, and deployment-request routes live in **API Reference** under **Model Catalog**. | Model | Context | Input | Output | Cached input | Vision | | --- | --- | --- | --- | --- | --- | | `moonshotai/Kimi-K3` | 1M | $3.00 | $15.00 | $0.45 | ✓ | | `zai-org/GLM-5.3` | 1M | $1.20 | $4.20 | $0.26 | — | | `deepseek-ai/DeepSeek-V4.1-Flash` | 1M | $0.50 | $1.50 | $0.05 | ✓ | | `zai-org/GLM-5.3-Flash` | 1M | $0.20 | $0.60 | $0.07 | ✓ | | `deepseek-ai/DeepSeek-V4-Flash-0731` | 1M | $0.28 | $0.56 | $0.07 | — | Coming soon: DeepSeek V4 Pro, Gemma 4 31B, OpenAI GPT-OSS 120B, NVIDIA Nemotron 3 Super, and Qwen3.8. Rates are confirmed per model at account provisioning; the site shows USD with an EUR toggle. ### Decisions models Served on [`POST /v1/decisions`](https://sference.com/docs/guides/decisions) (realtime only). It answers up to 64 choice, score, or yes/no questions about one input in a single call, generates no text, and is billed on input tokens only. | Model | Context | Input | Output | | --- | --- | --- | --- | | `Cloudflare/clef` | 64K | $0.24 | — | ## Auto model routing > **Beta** > > Auto model routing is in beta and **off by default**. An organization admin turns it on per organization (console → **Members** → the Organization card → **Auto model routing**, or `PATCH /control/v1/teams/me/organization {"auto_routing_enabled": true}`). Until then the model is hidden from every listing and requests for it return **403**. `sference/auto` picks a model for you, once per session, based on how hard the first message looks. A small classifier reads the first user message and rates it **low**, **medium**, or **high**; that tier's model then serves every later turn of the session. | Tier | Model | Images | | --- | --- | --- | | **low** | `deepseek-ai/DeepSeek-V4.1-Flash` | yes | | **medium** | `zai-org/GLM-5.3` | no | | **high** | `moonshotai/Kimi-K3` | yes | Send it like any other model id on `POST /v1/chat/completions`, `/v1/messages`, or `/v1/responses`: ```bash curl https://api.sference.com/v1/chat/completions \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"sference/auto","messages":[{"role":"user","content":"Refactor this module to use dependency injection."}]}' ``` ### What you get back The response `model` is the **routed** model, never `sference/auto` — so cost display and logging work without changes. Every response also carries two headers: | Header | Example | Meaning | | --- | --- | --- | | `x-sference-routed-model` | `moonshotai/Kimi-K3` | The model that served the request | | `x-sference-route-tier` | `high` | The tier it was routed to | You are billed at the **routed model's** rates. The classification call is not billed to you. ### How a session is identified The API is stateless and clients send no session id, so the router infers the session from how agent harnesses behave: **they resend the whole conversation on every request**. Turn 1 sends `[system, user1]`; turn 5 sends `[system, user1, …, user5]`. The start of the conversation is identical every time. A session is therefore every request sharing the same **team + endpoint + system prompt text + first user message text**. The router hashes those four, looks the hash up, and either decides (first time) or reuses the stored tier. The decision expires after **1 hour idle**, refreshed by every request in the session. Only the hash and tier are stored, never content. > **Where this can surprise you** > > - **Weak openers.** A session that starts with "hi", "continue", or `/init` is rated on that text, and the whole session stays on the low tier even if the real task arrives on turn 2. Start a session with the actual task. > - **System prompt drift.** If your harness puts something that changes into the system prompt (a date, git status, tool list), the session key changes mid-conversation and the session is re-decided — possibly switching models and losing its prompt cache. > - **Compaction.** `/compact` in Claude Code rewrites the first message, which produces a new decision. The cache is gone at that point anyway. > - **Subagents** start their own message history, so each gets its own decision. Usually what you want: an exploration subagent can run on the cheap tier. ### Limits - **Three endpoints only.** `POST /v1/chat/completions`, `/v1/messages`, and `/v1/responses`. Batches reject it at create with **400** — batch rows are not sessions. - **No cross-tier failover.** If the routed model is at capacity you get the normal 429/503 for that model. Silently changing tier mid-session would hide capacity problems and break the prompt cache. - **Images never go to the medium tier**, because `GLM-5.3` has no vision. An image on a medium session re-routes it to the high tier once. - **An org model allowlist still applies.** If it excludes `sference/auto`, requests get **403**; the router only routes to tiers the allowlist permits. ## Reading the catalog from the API `GET /v1/models` returns each entry with its capabilities and rates, so you can route on capability instead of hardcoding a list: ```bash curl https://api.sference.com/v1/models \ -H "Authorization: Bearer $SFERENCE_API_KEY" ``` ```json { "id": "zai-org/GLM-5.3-Flash", "display_name": "GLM 5.3 Flash", "context_tokens": 1048576, "capabilities": { "image_input": { "supported": true }, "pdf_input": { "supported": false }, "tools": { "supported": true }, "thinking": { "supported": true } }, "pricing": { "input_per_million_usd": 0.2, "cached_input_per_million_usd": 0.07, "output_per_million_usd": 0.6 } } ``` > **Capability comes from the catalog, not the model name** > > A model that does not support an input **rejects** it rather than silently dropping it. Check `capabilities.image_input.supported` before routing image traffic — see [Images and vision](https://sference.com/docs/guides/vision). ## Processing modes Every catalog model runs in all three [processing modes](https://sference.com/docs/api/responses#processing-modes): **realtime** for interactive traffic, **flex** for discounted latency-tolerant sync work, and **batch** (24h window) for background pipelines. Mode is a per-request choice, not a per-model one. ## Batch jobs `POST /v1/batches` validates every row's `model` against the catalog at create time and **rejects a batch that mixes models**. Use one model id per batch job. ## Bring your own model If it runs on **vLLM** or **SGLang**, we can usually serve it — fine-tunes, distilled checkpoints, and internal models run on the same API, the same audit trail, and the same processing modes as catalog models. Models that need a custom runtime are stood up per customer. Custom model hosting is arranged as part of an enterprise agreement; start at [hello@sference.com](mailto:hello@sference.com) or see [Custom models](https://sference.com/docs/custom-models). - [Pricing](https://sference.com/docs/pricing): Rates per model, the three windows, and what counts as a token. - [Custom models](https://sference.com/docs/custom-models): Bring your own Hugging Face weights. - [Support matrix](https://sference.com/docs/support): Which surfaces and fields each model supports. - [API reference](https://sference.com/docs/api-reference): Catalog, search, and deployment routes. --- # Pricing > How sference bills inference: per-token rates for every catalog model across the realtime, flex, and batch windows, cached input, and what counts as a token. Source: https://sference.com/docs/pricing sference bills **per token**. There is no platform fee, no seat cost, and no minimum spend on serverless: you pay for prompt tokens and completion tokens. Dedicated capacity is billed per GPU-hour instead — see [Deployments](https://sference.com/docs/deployments). All rates are **USD per 1M tokens**. The site shows the same numbers with a EUR toggle. ## Three windows, three prices Every request runs on one of three scheduling windows. The window decides how long the request may wait behind other work, and therefore the rate. | Window | How you select it | Wait budget | Rate | | --- | --- | --- | --- | | **realtime** | default; sync `/v1/chat/completions`, `/v1/messages`, or blocking `/v1/responses` | none — served as soon as capacity allows | list | | **flex** | `service_tier: "flex"` on `/v1/chat/completions` or `/v1/responses` | up to 30 minutes, best effort | **0.7 × list** | | **batch** | `background: true`, batches, or streams (`completion_window: "24h"`) | up to 24 hours | **0.5 × list** | Realtime, flex, and batch are **scheduling priorities on the same fleet**, not separate deployments. Interactive traffic never queues behind backfill, which is what makes the discounted windows possible without degrading realtime. > **Flex is per-account** > > `service_tier: "flex"` must be enabled on your account. Ask your account team to turn it on. A flex request that cannot complete inside its window fails with `408`, and the OpenAI SDKs retry it automatically. Flex and batch are **independent dimensions** and they stack: a batch request submitted with `service_tier: "flex"` bills at `0.7 × 0.5 = 0.35 ×` list. ## Catalog rates USD per 1M tokens. **Cached input** is the rate for prompt tokens served from the prefix cache — see [Prompt caching](https://sference.com/docs/guides/prompt-caching). | Model | Window | Input | Cached input | Output | | --- | --- | ---: | ---: | ---: | | `moonshotai/Kimi-K3` | realtime | 3.00 | 0.45 | 15.00 | | | flex | 2.10 | 0.315 | 10.50 | | | batch | 1.50 | 0.225 | 7.50 | | `zai-org/GLM-5.3` | realtime | 1.20 | 0.26 | 4.20 | | | flex | 0.84 | 0.182 | 2.94 | | | batch | 0.60 | 0.13 | 2.10 | | `zai-org/GLM-5.2` | realtime | 1.20 | 0.26 | 4.20 | | | flex | 0.84 | 0.182 | 2.94 | | | batch | 0.60 | 0.13 | 2.10 | | `deepseek-ai/DeepSeek-V4.1-Flash` | realtime | 0.50 | 0.05 | 1.50 | | | flex | 0.35 | 0.035 | 1.05 | | | batch | 0.25 | 0.025 | 0.75 | | `deepseek-ai/DeepSeek-V4-Flash-0731` | realtime | 0.28 | 0.07 | 0.56 | | | flex | 0.196 | 0.049 | 0.392 | | | batch | 0.14 | 0.035 | 0.28 | | `deepseek-ai/DeepSeek-V4-Flash` | realtime | 0.28 | 0.07 | 0.56 | | | flex | 0.196 | 0.049 | 0.392 | | | batch | 0.14 | 0.035 | 0.28 | | `Cloudflare/clef` | realtime | 0.24 | — | 0.00 | | | flex | 0.168 | — | 0.00 | | | batch | 0.12 | — | 0.00 | | `zai-org/GLM-5.3-Flash` | realtime | 0.20 | 0.07 | 0.60 | | | flex | 0.14 | 0.049 | 0.42 | | | batch | 0.10 | 0.035 | 0.30 | `Cloudflare/clef` has no output charge — it is a decisions model and bills on input only. > **This table is a snapshot** > > `GET https://api.sference.com/v1/models` is authoritative and reflects your entitlements, including custom models. Rates for models agreed in your contract are per contract. Read the API at runtime rather than hardcoding this table. ## What counts as a token - **Input** — every token in the prompt, including system messages, tool definitions, and images converted to tokens. Tokens served from the prefix cache bill at the **cached input** rate instead. - **Output** — every generated token, **including reasoning tokens**. A model that thinks for 2,000 tokens before answering 100 bills 2,100 output tokens. This is why `max_output_tokens` matters on reasoning models: the cap covers the thinking too. - **No cache-write charge.** Unlike some providers, sference does not bill a premium for writing into the prefix cache, and no TTL choice changes the price. Cached tokens are simply cheaper. - **Rejected requests are not billed.** A `max_queue_wait_exceeded` rejection, a `408` on flex, or an upstream failure before any output costs nothing. A request that produced output and then failed bills for the tokens it produced. ## Checking what you spent | Surface | What it shows | | --- | --- | | `GET /control/v1/billing/summary` | Current period spend, balance, and burn rate | | `GET /control/v1/usage/summary` | Token and cost totals, optionally grouped by model, key, or member | | `GET /control/v1/activity` | Per-request rows: model, tokens, cost, latency, outcome | See [Usage & analytics](https://sference.com/docs/usage) for the full endpoint list, CSV exports, and attribution. ## Dedicated and enterprise - **Dedicated deployments** — reserved GPU capacity with private endpoints, billed per GPU-hour on monthly commitments. Predictable latency and no per-token metering. See [Deployments](https://sference.com/docs/deployments). - **Custom models** — fine-tunes and distilled models are hosted under an enterprise agreement; pricing is per contract. See [Custom models](https://sference.com/docs/custom-models). - **Enterprise** — annual agreements, committed-spend discounts, volume pricing, and custom SLAs. Contact `hello@sference.com`. ## Why per-token beats self-managed The per-token rate is only comparable to self-hosting if your GPUs are busy. Our scheduler keeps utilization high around the clock by packing realtime traffic and batch backfill onto the same fleet; on self-managed instances or an on-prem cluster, idle capacity is paid for whether it serves tokens or not. The rate you see is the cost of the tokens, not the cost of the reservation. - [Pricing for agents](https://sference.com/docs/pricing-for-agents): The same rates as a compact Markdown table, for agents and exports. - [Prompt caching](https://sference.com/docs/guides/prompt-caching): Cached input is the largest lever on most bills. - [Usage & analytics](https://sference.com/docs/usage): Attribute spend to a workload, key, or team member. --- # Usage & analytics > Query spend, balance, token usage, and per-request activity programmatically — grouped by model, key, member, and department. Source: https://sference.com/docs/usage Everything the console shows about spend is available over the API, under `/control/v1`. Use it to build your own dashboards, alert on burn rate, or attribute cost to a workload without instrumenting your application. ## The three levels | Level | Endpoint | Answers | | --- | --- | --- | | **Summary** | `GET /control/v1/usage/summary` | How many tokens and dollars, this period | | **Grouped** | `GET /control/v1/usage/by-model`, `/by-key`, `/by-member`, `/by-department` | Where the spend went | | **Per request** | `GET /control/v1/activity` | Which request, what it cost, how it went | ## Summary ```bash curl "https://api.sference.com/control/v1/usage/summary?from=2026-10-01&to=2026-10-31" \ -H "Authorization: Bearer $SFERENCE_API_KEY" ``` Returns token totals, cost, and request counts for the period. ## Grouped breakdowns ```bash # Spend by model curl "https://api.sference.com/control/v1/usage/by-model?from=2026-10-01" \ -H "Authorization: Bearer $SFERENCE_API_KEY" # Spend by API key — the per-engineer view for vibe coding curl "https://api.sference.com/control/v1/usage/by-key?from=2026-10-01" \ -H "Authorization: Bearer $SFERENCE_API_KEY" # Spend by team member curl "https://api.sference.com/control/v1/usage/by-member?from=2026-10-01" \ -H "Authorization: Bearer $SFERENCE_API_KEY" # Spend by department (for chargeback) curl "https://api.sference.com/control/v1/usage/by-department?from=2026-10-01" \ -H "Authorization: Bearer $SFERENCE_API_KEY" ``` Each has a CSV twin for spreadsheets and finance tooling: ``` /control/v1/usage/by-model.csv /control/v1/usage/by-key.csv /control/v1/usage/by-member.csv /control/v1/usage/by-department.csv ``` ### Daily series `GET /control/v1/usage/daily` returns a per-day series for charting — the endpoint to poll for a burn-rate dashboard or an anomaly alert. ## Per-request activity `GET /control/v1/activity` is the request-level log: model, token counts, cost, latency, and outcome, one row per request. `GET /control/v1/activity/{request_id}` returns a single request in detail, and `GET /control/v1/activity/models` lists the models that appear in your activity. ```bash curl "https://api.sference.com/control/v1/activity?limit=100" \ -H "Authorization: Bearer $SFERENCE_API_KEY" ``` `/control/v1/activity/export.csv` streams the whole log as CSV for offline analysis. > **Attribution comes from the key, not the payload** > > Every request is attributed to the API key that made it. That is why one key per engineer, per service, or per workload is the simplest way to get useful breakdowns — no application instrumentation, no tagging in the request body. ## Billing | Endpoint | Returns | | --- | --- | | `GET /control/v1/billing/summary` | Current period spend, balance, and burn rate | | `GET /control/v1/billing/balance` | Available balance | | `GET /control/v1/billing/transactions` | Ledger entries — charges, top-ups, credits | | `GET /control/v1/billing/topups` | Top-up history | ## A burn-rate alert ```python import requests def check_burn_rate(threshold_usd: float = 50.0): r = requests.get( "https://api.sference.com/control/v1/usage/daily", headers={"Authorization": f"Bearer {SFERENCE_KEY}"}, params={"from": today_minus(7)}, ) days = r.json()["data"] avg = sum(d["cost_usd"] for d in days) / len(days) if avg > threshold_usd: alert(f"sference burn rate is ${avg:.2f}/day, above ${threshold_usd:.2f}") ``` ## Streams A [stream](https://sference.com/docs/guides/streams) rolls up its own tokens and cost, which is the cleanest attribution when a workload spans many requests: ``` GET /v1/streams/{stream_id} ``` `total_cost_micros` on the stream detail is the workload's spend in millionths of a dollar. - [Pricing](https://sference.com/docs/pricing): Rates, windows, and what counts as a token. - [Streams](https://sference.com/docs/guides/streams): Per-workload cost rollups. - [SferenceCode](https://sference.com/docs/sference-code): Per-engineer attribution for coding agents. --- # Sference Switch > A macOS app that routes every Claude Code and Codex session on your Mac to sference models with one switch, and back to native providers with the same switch. Source: https://sference.com/docs/switch **Sference Switch** is a macOS menu bar app with a local gateway. It routes the coding agents you already use, Claude Code and Codex, between their native providers and open models served by sference. It is controlled by a single on/off switch. [`sference launch`](https://sference.com/docs/cli#launch-a-coding-agent) configures one session at a time. Switch works at the machine level instead: once it is on, every Claude Code session on the Mac is routed, including ones that are already running. You don't need a wrapper command, environment variables, or edits to Claude Code's configuration. ![The Sference Switch app: the Switch toggle, the signed-in account, and the live request path for Claude Code and Codex](https://sference.com/docs/sference-switch-overview.png) > **Beta** > > Sference Switch is under active development, and its interfaces and configuration may change between releases. It requires macOS 13 or newer (Apple Silicon or Intel). The build is not yet notarized by Apple, so on first launch you may need to allow it under **System Settings → Privacy & Security → Open Anyway**. ## Install ```bash curl -fsSL https://get.sference.com | sh ``` The installer downloads the latest release, verifies its checksum, installs the `sference-switch` CLI to `~/.local/bin` and the app to `~/Applications`. Homebrew is not needed. ## Get started Open the app, then: 1. **Sign in.** Go to **Account → Sign In**. The app shows a verification code and opens the approval page in your browser. 2. **Turn on the Switch.** The first time, macOS asks for your password once. This installs a local system service and adds an entry to `/etc/hosts`. 3. **Pick a model.** In Claude Code, run `/model`. Sference models are listed with a `[Sference]` prefix next to the Claude models. To use a 1M-context model's full window, choose its `… (1M context)` entry. Claude models keep working as before. Anything you don't route to sference goes to Anthropic with your normal login. ## What the app shows - **Switch**: global routing on/off. Your model mappings are kept while it is off. - **Account**: the signed-in sference account and session state. - **Live request path**: for each client, the gateway's health, the model that is actually serving requests, and the fallback. - **Traffic** and **Requests**: recent requests, latency, and spend. - **Clients**: per-client model mapping for Claude Code and Codex. ## Claude Code Claude Code needs no configuration and no restart. When the Switch is on, inference requests (`/v1/messages`) and model discovery go to sference. Sign-in, feature flags, and everything else go to Anthropic unchanged, so Claude Code behaves exactly as it does natively. You can also map whole Claude model families to a sference model, for example so that sub-agents requesting `sonnet` run on GLM: ```bash sference-switch claude route sonnet zai-org/GLM-5.2 sference-switch claude route sonnet native # send this family back to Anthropic sference-switch claude subagents zai-org/GLM-5.2 ``` A family can also be mapped to `sference/auto` ([auto model routing](https://sference.com/docs/models#auto-model-routing), beta, org opt-in), which picks a model per session instead of pinning one: ```bash sference-switch claude route sonnet sference/auto ``` ## Codex Codex support is opt-in. Switch writes a separate profile (`~/.codex/sference.config.toml`) and does not modify `~/.codex/config.toml`: ```bash sference-switch codex on sference-switch codex route zai-org/GLM-5.2 codex --profile sference ``` Start Codex without `--profile sference` to use OpenAI as usual. `sference-switch codex off` removes the profile. ## Turning it off Flip the Switch off in the app, or run `sference-switch off`. Requests go back to native providers immediately. To remove the system service and the `/etc/hosts` entry as well: ```bash sudo sference-switch intercept off sudo sference-switch tls service uninstall ``` `sference-switch uninstall --dry-run` shows everything a full uninstall would remove. ## How it works Turning the Switch on points `api.anthropic.com` at a local service via `/etc/hosts`. That service presents a certificate from a local Sference Switch certificate authority, which is trusted in your System keychain. It forwards inference and model-list requests to the sference gateway and passes all other traffic through to Anthropic untouched. Turning the Switch off removes the redirect. - **Everything stays on loopback.** The gateway and its admin API bind only to `127.0.0.1`. - **Credentials go only to their own provider.** Your sference credential is sent only to sference, and your Anthropic or OpenAI credentials only to Anthropic or OpenAI. - **Telemetry is local, metadata only.** Records contain no prompts, responses, headers, or credentials, and you can turn them off in `~/.sference/switch/gateway.yaml`. ## Troubleshooting ```bash sference-switch status sference-switch doctor --probe ``` `doctor` checks the whole request path and prints the first failure with a concrete fix. Logs are in `~/.sference/switch/logs/`. If your session expires, sign in again from the app or with `sference-switch auth login`. Source, the full command reference, and releases: **[github.com/s-ference/sference-switch](https://github.com/s-ference/sference-switch)** (MIT). - [sference launch](https://sference.com/docs/cli#launch-a-coding-agent): Run Claude Code, Codex, opencode, or Pi on sference for one session. - [Claude Code](https://sference.com/docs/sference-code/claude-code): Model recommendations, known limits, and manual configuration. --- # Python SDK > Install sference-sdk, Responses API patterns, framework examples (Prefect), and agent prompts for Cursor and Claude Code. Source: https://sference.com/docs/sdk The **`sference-sdk`** package on PyPI is the supported way to call sference from Python. Source and API contract live in the **[sference OSS repository](https://github.com/s-ference/sference)**. ## Install ```bash pip install sference-sdk ``` ```bash uv add sference-sdk ``` Import name: **`sference_sdk`**. ## Credentials ```bash export SFERENCE_API_KEY='sk_...' ``` Pass the key in code when you prefer not to use the environment: `SferenceClient(api_key="sk_...")`. > **Picking a pattern** > > For interactive code, anything where a user is waiting, call the blocking response/chat methods; that is the common case and the one the platform is tuned for. For unattended jobs, use **`create_response(..., background=True)`** then **`wait_for_response`**, which runs in the background with no blocking HTTP connection. The SDK mirrors the API's [three processing modes](https://sference.com/docs/api/responses#processing-modes). ## Example (Responses API) ```python import os from sference_sdk import SferenceClient client = SferenceClient(api_key=os.environ["SFERENCE_API_KEY"]) resp = client.create_response( model="deepseek-ai/DeepSeek-V4-Flash-0731", input=[{"role": "user", "content": "Say hello in one sentence."}], background=True, metadata={"completion_window": "24h"}, # "24h" is the only supported value ) done = client.wait_for_response(resp.id, poll_interval=2.0, timeout=3600.0) print(done.status) ``` For realtime classification, `client.create_decision(model="Cloudflare/clef", state=..., questions=...)` answers choice, score, and yes/no questions in one call. See **[Decisions](https://sference.com/docs/guides/decisions)**. ## Processing modes and completion windows | Mode | Where | Window | |------|-------|--------| | **Realtime** | Sync chat, [Anthropic messages](https://sference.com/docs/api/messages), blocking `/v1/responses` | No window; returns when inference finishes | | **Flex** | `/v1/chat/completions` or `/v1/responses` with `service_tier: "flex"` (flex-enabled accounts; not `/v1/messages`) | No window; discounted, lower scheduling priority, raise client timeouts | | **Async** | Background responses, streams, batches | `"24h"` (the only supported value) | Set the batch window via `metadata={"completion_window": "24h"}` on background responses, `window=` on `create_stream` / `submit_batch`. For **grouped work** (e.g. one preset for log analysis), create a **stream** first, then pass `metadata={"stream_id": stream.id, "completion_window": "24h"}` on each response. For a fixed JSONL file run once, use **`submit_batch`**. Each row shares one **catalog-available model** (same id on every line, or pass `model=` for content-only JSONL). Row `body` may use chat `messages` or Responses `input` (normalized at create). See **[Batch inference](https://sference.com/docs/guides/batches)**. ```python batch = client.submit_batch( input_file="./workload.jsonl", model="deepseek-ai/DeepSeek-V4-Flash-0731", window="24h", ) done = client.wait_for_completion(batch.id, poll_interval=5.0, timeout=86_400.0) client.download_results_jsonl(done.id, out="./results.jsonl") ``` See [OpenAI Responses API](https://sference.com/docs/api/responses) for the HTTP-oriented guides. ## Framework examples Runnable workflows in the **[sference OSS repo](https://github.com/s-ference/sference/tree/main/examples)** show how to combine **`sference-sdk`** with orchestration tools. Clone that repo (or copy the example into your project), install example deps, and set `SFERENCE_API_KEY`. ```bash git clone https://github.com/s-ference/sference.git cd sference uv sync --group dev --group examples export SFERENCE_API_KEY='sk_...' export SFERENCE_MODEL='deepseek-ai/DeepSeek-V4-Flash-0731' # model id from your deployment ``` More frameworks will be added here as examples land in `examples/`. ### Prefect **[examples/prefect/](https://github.com/s-ference/sference/tree/main/examples/prefect)**: two-stage flow for batch-style analysis: enqueue background responses, then poll to completion. Adapted from Prefect’s [AI data analyst example](https://github.com/PrefectHQ/prefect/blob/main/examples/ai_data_analyst_with_pydantic_ai.py), using sference instead of an in-process LLM. | Stage | Prefect | SDK | |-------|---------|-----| | Prepare | `@task`: build prompts from local data (pandas, files) | No API calls | | Submit | `@task` + `.map(prompts)` | `create_response(..., background=True, metadata={"completion_window": "24h"})` → `response.id` | | Wait | `@task` + `.map(response_ids)` | `wait_for_response(id)` | | Report | `@task` | Parse `done.output` (`message` → `output_text` parts) | **Why split submit and wait?** Prefect retries and observability stay in the orchestrator; inference runs on sference. If a **wait** task fails, you can retry polling without re-submitting jobs that already have an ID. Run locally: ```bash uv run python examples/prefect/ai_data_analyst_batch_responses.py ``` Deploy with Prefect (worker + UI): ```bash uv run python examples/prefect/ai_data_analyst_batch_responses.py --serve ``` Then trigger **ai-data-analyst-batch-responses** from the Prefect UI or CLI. Source and setup notes: **[examples/prefect/README.md](https://github.com/s-ference/sference/blob/main/examples/prefect/README.md)** · **[ai_data_analyst_batch_responses.py](https://github.com/s-ference/sference/blob/main/examples/prefect/ai_data_analyst_batch_responses.py)**. ## Coding agents (Cursor, Claude Code, …) Two layers in the OSS repo: use the **prompt** for vibe coding; add the **skill file** when you want depth or a Cursor skill install. > **Teaching an agent the SDK vs. running the agent on sference** > > This section teaches your coding agent to **write code against the sference SDK**. If you instead want Claude Code itself to **run on sference models**, that is the [Anthropic Messages endpoint](https://sference.com/docs/api/messages); see [Claude Code](https://sference.com/docs/sference-code/claude-code). ### 1. Agent prompt (copy this first) Canonical file (always current on `main`): **[github.com/s-ference/sference/blob/main/PROMPT.txt](https://github.com/s-ference/sference/blob/main/PROMPT.txt)** Raw URL to open and copy: ```text https://raw.githubusercontent.com/s-ference/sference/main/PROMPT.txt ``` Paste into **Cursor** project rules, **Claude Code** `CLAUDE.md`, or your agent’s custom instructions. It points to the full skill file for examples and edge cases. ### 2. Full skill (optional) For Cursor’s skill format, copy **[SKILL.md](https://github.com/s-ference/sference/blob/main/SKILL.md)** (keep the YAML frontmatter) into your app repo: ```text .cursor/skills/sference-sdk/SKILL.md ``` Raw: `https://raw.githubusercontent.com/s-ference/sference/main/SKILL.md` Install steps for every tool: **[AGENTS.md](https://github.com/s-ference/sference/blob/main/AGENTS.md)** in the OSS repo. ## Go deeper - [Framework examples](https://github.com/s-ference/sference/tree/main/examples): Prefect today; more orchestration samples in examples/. - [SDK README](https://github.com/s-ference/sference/blob/main/sdk-python/README.md): Batch vs stream, the async client, and the method reference in the OSS repo. - [Batch inference](https://sference.com/docs/guides/batches): JSONL jobs, body shapes, and create-time validation. - [OpenAI Responses API](https://sference.com/docs/api/responses): HTTP guide for background responses and stream presets. - [CLI](https://sference.com/docs/cli): One-line install for the sference command. --- # CLI > Install the sference CLI, authenticate, and launch Claude Code, Codex, opencode, or Pi on sference models. Source: https://sference.com/docs/cli The **`sference`** command-line tool is published on PyPI as **`sference-cli`**. It wraps the same inference API as the Python SDK (auth, batches, streams, and responses), and `sference launch` runs your coding agent on sference models. ## Install (one line) macOS and Linux: ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh ``` That script installs **`sference`** on your PATH (via `uv tool` when available). ## Other install options ```bash uv tool install sference-cli ``` ```bash pip install sference-cli # or pipx install sference-cli ``` Then verify: ```bash sference --help ``` ## Authenticate Sign in with your sference account. No API key needed: ```bash sference auth login ``` The CLI opens the [sference console](https://app.sference.com) in your browser and shows a short code. Check that it matches the code in the console, approve the device, and the CLI finishes signing in on its own. Use `--no-browser` on a remote or headless machine: the CLI prints the approval URL and code, and you open the URL on any device where you're signed in. The CLI saves a 24-hour access token and a 30-day refresh token to `~/.sference/credentials.json` and refreshes the access token automatically whenever you run a command. `sference launch claude` also refreshes it mid-session, so a long Claude Code session doesn't drop. Codex, opencode, and Pi sessions run on the token issued at launch, which is valid for 24 hours. You only need to sign in again if you go 30 days without using the CLI or revoke the device. Each machine is listed under **Profile → Signed-in devices** in the console, where you can revoke it. Check who you're signed in as: ```bash sference auth me ``` ### CI and scripts: API key For non-interactive environments, create an API key in the [console](https://app.sference.com) and either save it or set it in the environment: ```bash sference auth login --api-key 'sk_...' # saves the key, no browser export SFERENCE_API_KEY='sk_...' # or: overrides any saved credential for every subcommand ``` ## Launch a coding agent `sference launch` runs the coding agent you already use on open models served by sference. One command configures the agent and starts it; your existing settings, logins, and sessions are left as they are. ```bash sference launch claude # Claude Code sference launch codex # OpenAI Codex CLI sference launch opencode # opencode sference launch pi # Pi ``` Every launcher works the same way: - **Your sference account, nothing else.** The launcher uses your `sference auth login` sign-in (or `SFERENCE_API_KEY`). Usage is billed to your account like any other request. - **Every catalog model in the picker.** The launcher fetches the live catalog from `GET /v1/models`, so each model shows up in the agent's own model picker (`/model` and similar). - **Pick a default.** `--model ` sets the model the session starts on. Without it the launcher uses `SFERENCE_MODEL`, then `zai-org/GLM-5.2`. `--model sference/auto` starts the session on [auto model routing](https://sference.com/docs/models#auto-model-routing) (beta, org opt-in), and the picker lists **Sference Auto** alongside the catalog models. - **Forward agent flags after `--`.** For example, `sference launch codex -- exec "fix the failing test"`. - **See what it does first.** `--dry-run` prints the configuration and the command without starting the agent, with the API key redacted. For opencode and Pi it still writes the provider config file. - **Going back is just starting the agent normally.** Run `claude`, `codex`, `opencode`, or `pi` the usual way to use its native models. Claude Code and Codex keep no sference configuration. opencode and Pi keep a `sference` provider entry, which sits alongside your other providers until you pick it. | Agent | What `sference launch` configures | | --- | --- | | **Claude Code** | Starts a local proxy for this session only. sference models appear in `/model` next to Claude models, and Claude models still reach Anthropic through your normal login, so you can use both in one session. `--no-anthropic` sends everything to sference via `ANTHROPIC_BASE_URL` instead. Full guide: [Claude Code](https://sference.com/docs/sference-code/claude-code). | | **Codex CLI** | Adds a `sference` provider on sference's `/v1/responses` through `codex -c` overrides, so `~/.codex/config.toml` is never modified. Sub-agents and MCP servers run on sference too, and auto-compaction triggers at the model's real context window. | | **opencode** | Merges a `sference` provider into `~/.config/opencode/opencode.json`, with a thinking-effort variant on every model. Your other providers, keybinds, and theme are preserved. | | **Pi** | Writes a `sference` provider to `~/.pi/agent/models.json` (other providers are preserved) and scopes Pi's model picker to sference models. Reasoning effort maps to Pi's native levels. | The agent itself must already be installed and on your `PATH` (`claude`, `codex`, `opencode`, or `pi`). Claude Code's proxy mode also needs `mitmproxy`, which the installer above sets up. > **Want it on for every session?** > > `sference launch` scopes sference to the session it starts. To route the Claude Code and Codex sessions you start yourself, without a launcher, use [Sference Switch](https://sference.com/docs/switch): a macOS app with a single on/off switch. ## Batch JSONL Each line in a batch input file becomes one row in `POST /v1/batches`. **All rows must use the same model**: either repeat `body.model` on every OpenAI-style line or set **`--model`** once for content-only lines. The model must be catalog-available; invalid or mixed models return **400** at create. Inner `body` accepts **chat completions** (`messages`) or **Responses** (`input`). Responses fields are normalized at create; invalid rows return **400** with row index / `custom_id`. ```jsonl {"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"…","messages":[{"role":"user","content":"hi"}]}} {"custom_id":"b","method":"POST","url":"/v1/responses","body":{"model":"…","input":[{"role":"user","content":"hi"}]}} {"content":"content-only, pass --model on submit"} ``` Full guide: **[Batch inference](https://sference.com/docs/guides/batches)**. Subcommand reference: **[OSS CLI README](https://github.com/s-ference/sference/blob/main/cli/README.md)**. ## Decisions `sference decisions create` sends one realtime [decision](https://sference.com/docs/guides/decisions). It prints one line per answer, or the full response with `--json`: ```bash sference decisions create --model Cloudflare/clef \ --state "I was charged twice, please refund me" \ --questions '{"route": {"type": "choice", "criteria": {"billing": "Payments", "support": "Other"}}, "refund": {"type": "noul"}}' ``` ## Go deeper Subcommands, JSONL batch formats, and stream workflows live in the **[sference OSS repository](https://github.com/s-ference/sference)** (`cli/README.md`). - [Sference Switch](https://sference.com/docs/switch): Route every Claude Code and Codex session on your Mac with one switch. - [Python SDK](https://sference.com/docs/sdk): Library-first path for application code and coding agents. - [Quickstart](https://sference.com/docs/quickstart): API keys and your first inference request. --- # LLM & agent access > Read these docs by machine: Markdown views, llms.txt, llms-full.txt, and the OpenAPI specification. Source: https://sference.com/docs/llm-access Everything on this site is designed to be read by an LLM as easily as by a person. If you are wiring up a coding agent — or you *are* the coding agent — use the surfaces below instead of scraping HTML. ## Markdown views Any docs page is available as Markdown by appending `.md` to its URL: ```bash curl https://sference.com/docs/quickstart.md curl https://sference.com/docs/agents/langchain.md curl https://sference.com/docs/api-reference/inference-api/V1ChatCompletionsCreateChatCompletion.md ``` You can also ask for Markdown on the plain URL with content negotiation: ```bash curl -H "Accept: text/markdown" https://sference.com/docs/quickstart ``` Both work for guide pages and for every API-reference page. Blog posts have the same treatment (`/blog/.md`), as do the legal pages (`/legal/dpa.md`, `/legal/subprocessors.md`, `/privacy.md`, `/terms.md`). ## Indexes for one-shot loading | File | Contents | | --- | --- | | [`/llms.txt`](https://sference.com/llms.txt) | Curated index: what sference is, capabilities, pricing, and a link per docs page. The right entry point for an agent deciding what to read. | | [`/llms-full.txt`](https://sference.com/llms-full.txt) | Every prose page concatenated into one file, with each page's Markdown twin linked. For loading the whole documentation into context at once. | | [`/openapi.json`](https://sference.com/openapi.json) | The machine-readable API specification behind the API reference. | ```bash # Load the whole documentation curl https://sference.com/llms-full.txt # Or just the index, then fetch what you need curl https://sference.com/llms.txt ``` ## OpenAPI specification `https://sference.com/openapi.json` is the complete spec: every inference route, every control-plane route, and every schema. Point a code generator or an agent at it to produce a client without reading prose. ## A prompt for a coding agent Paste this into your agent to give it a working understanding of the API: ```text You are writing code against the sference inference API. - Base URL: https://api.sference.com/v1 - Auth: `Authorization: Bearer sk_...` (also accepts `x-api-key` on /v1/messages) - Surfaces: the OpenAI Chat API, the OpenAI Responses API, and the Anthropic Messages API - Model ids are catalog ids, e.g. "zai-org/GLM-5.3". There are no gpt-* or claude-* aliases. - Windows: realtime (default), flex (`service_tier: "flex"`, 0.7x), batch (`background: true` + `metadata.completion_window: "24h"`, 0.5x). - Set `prompt_cache_key` to a stable per-session id so the shared prefix bills at the cached rate. - Set `max_queue_wait_s` (5-300) to fail fast under congestion instead of hanging. - The API is stateless: send the full conversation every turn. `previous_response_id` and `conversation` are ignored. Read https://sference.com/llms.txt for the index, and append `.md` to any docs page URL for its Markdown source. ``` ## Crawling The site's `robots.txt` explicitly allows AI crawlers, training crawlers included. Public documentation in an LLM's weights is distribution, not leakage. - [Quickstart](https://sference.com/docs/quickstart): Your first request, in curl or the CLI. - [API compatibility](https://sference.com/docs/compatibility): What is honored, what is ignored. - [Python SDK](https://sference.com/docs/sdk): Library usage and agent prompts. --- # Pricing for agents > Public model prices as a compact Markdown table for agents and exports: every catalog model across the realtime, flex, and batch windows. Source: https://sference.com/docs/pricing-for-agents Use this table to read or compare sference's public list prices without parsing the visual [Pricing](https://sference.com/docs/pricing) page. All amounts are **USD per 1M tokens**. Model ids are case-sensitive. This table contains the same models and rates as [Pricing](https://sference.com/docs/pricing), covering **realtime**, **flex**, and **batch**. Hidden and restricted models are excluded. Both tables are generated from the same public catalog; this is not a live quote or a feed of organization-specific rates. ## Read or export No API key is required. Open this page as Markdown, or read the live catalog: ```bash # This page as Markdown curl https://sference.com/docs/pricing-for-agents.md # The authoritative live catalog (requires a key) curl https://api.sference.com/v1/models \ -H "Authorization: Bearer $SFERENCE_API_KEY" ``` The Markdown table is the published export. Any JSON or CSV you ask an agent to produce is a conversion of that table, not a separate API response. ## Public rates Each row identifies a model and a completion window. **Input** is uncached input; **cached input** is input served from the prompt cache; **output** includes generated reasoning tokens. See [Processing modes](https://sference.com/docs/api/responses#processing-modes) for scheduling behavior. | Model ID | Window | Input | Cached input | Output | | - | - | -: | -: | -: | | `Cloudflare/clef` | realtime | 0.24 | — | 0.00 | | `Cloudflare/clef` | flex | 0.168 | — | 0.00 | | `Cloudflare/clef` | batch | 0.12 | — | 0.00 | | `moonshotai/Kimi-K3` | realtime | 3.00 | 0.45 | 15.00 | | `moonshotai/Kimi-K3` | flex | 2.10 | 0.315 | 10.50 | | `moonshotai/Kimi-K3` | batch | 1.50 | 0.225 | 7.50 | | `zai-org/GLM-5.3` | realtime | 1.20 | 0.26 | 4.20 | | `zai-org/GLM-5.3` | flex | 0.84 | 0.182 | 2.94 | | `zai-org/GLM-5.3` | batch | 0.60 | 0.13 | 2.10 | | `zai-org/GLM-5.2` | realtime | 1.20 | 0.26 | 4.20 | | `zai-org/GLM-5.2` | flex | 0.84 | 0.182 | 2.94 | | `zai-org/GLM-5.2` | batch | 0.60 | 0.13 | 2.10 | | `deepseek-ai/DeepSeek-V4.1-Flash` | realtime | 0.50 | 0.05 | 1.50 | | `deepseek-ai/DeepSeek-V4.1-Flash` | flex | 0.35 | 0.035 | 1.05 | | `deepseek-ai/DeepSeek-V4.1-Flash` | batch | 0.25 | 0.025 | 0.75 | | `deepseek-ai/DeepSeek-V4-Flash-0731` | realtime | 0.28 | 0.07 | 0.56 | | `deepseek-ai/DeepSeek-V4-Flash-0731` | flex | 0.196 | 0.049 | 0.392 | | `deepseek-ai/DeepSeek-V4-Flash-0731` | batch | 0.14 | 0.035 | 0.28 | | `deepseek-ai/DeepSeek-V4-Flash` | realtime | 0.28 | 0.07 | 0.56 | | `deepseek-ai/DeepSeek-V4-Flash` | flex | 0.196 | 0.049 | 0.392 | | `deepseek-ai/DeepSeek-V4-Flash` | batch | 0.14 | 0.035 | 0.28 | | `zai-org/GLM-5.3-Flash` | realtime | 0.20 | 0.07 | 0.60 | | `zai-org/GLM-5.3-Flash` | flex | 0.14 | 0.049 | 0.42 | | `zai-org/GLM-5.3-Flash` | batch | 0.10 | 0.035 | 0.30 | ## Window factors | Window | Factor | How to select | | - | -: | - | | realtime | 1.0 × | default on `/v1/chat/completions`, `/v1/messages`, blocking `/v1/responses` | | flex | 0.7 × | `service_tier: "flex"` on `/v1/chat/completions` or `/v1/responses` | | batch | 0.5 × | `background: true`, batches, or streams with `completion_window: "24h"` | Flex and batch stack: a batch request with `service_tier: "flex"` bills at **0.35 ×** list. ## Notes for automated comparison - **Units are USD per 1M tokens.** Do not rescale. - **`—` in cached input** means the model has no prompt-cache rate; cached input is not billed separately for that model. - **`Cloudflare/clef` has no output charge** (0.00). It is a decisions model and bills on input only. - **Aliases bill at the rate of the model they resolve to.** `zai-org/GLM-5.2` resolves to `zai-org/GLM-5.3`; `deepseek-ai/DeepSeek-V4-Flash` resolves to `deepseek-ai/DeepSeek-V4-Flash-0731`. See [Models](https://sference.com/docs/models#aliases-and-retirement). - **Deprecated models are excluded** from this table. They remain callable at their existing rates. - **No cache-write charge.** Unlike some providers, there is no premium for writing into the prompt cache, and no TTL-dependent rate. - **Rejected requests are free.** A `max_queue_wait_exceeded` rejection or a `408` on flex is not billed. ## Verify against the live catalog Rates change. Before quoting these numbers to a user, confirm against the API: ```bash curl -s https://api.sference.com/v1/models \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ | jq '.data[] | {id, pricing}' ``` - [Pricing](https://sference.com/docs/pricing): The visual pricing page: windows, what counts as a token, and how to check spend. - [Models](https://sference.com/docs/models): Capabilities, context windows, aliases, and retirement. - [LLM & agent access](https://sference.com/docs/llm-access): Markdown views, llms.txt, and the OpenAPI spec. --- # API compatibility > What each sference surface accepts, which request fields are honored, and which are ignored. Source: https://sference.com/docs/compatibility sference speaks three request shapes: the OpenAI Chat Completions API, the OpenAI Responses API, and the Anthropic Messages API. This page is the honest map: what each surface accepts, what we honor, and what we ignore. ## Surfaces | Surface | Endpoint | Shape | | --- | --- | --- | | OpenAI Chat API | `POST /v1/chat/completions` | OpenAI | | OpenAI Responses API | `POST /v1/responses` | OpenAI | | Anthropic Messages API | `POST /v1/messages` | Anthropic | | Embeddings | `POST /v1/embeddings` | OpenAI | | Batches | `POST /v1/batches` | OpenAI | | Files | `POST /v1/files` | OpenAI | | Streams | `POST /v1/streams` | sference extension | | Models | `GET /v1/models` | OpenAI | ## Field support ### OpenAI Chat API | Field | Status | | --- | --- | | `model`, `messages`, `stream` | ✅ | | `temperature`, `top_p`, `top_k`, `min_p` | ✅ | | `max_tokens`, `max_completion_tokens` | ✅ | | `stop`, `seed` | ✅ | | `presence_penalty`, `frequency_penalty`, `repetition_penalty` | ✅ | | `tools`, `tool_choice` | ✅ | | `response_format` (`json_schema`, `json_object`) | ✅ — see [Structured outputs](https://sference.com/docs/guides/structured-outputs) | | `stream_options.include_usage` | ✅ | | `reasoning_effort`, `enable_thinking` | ✅ on reasoning models | | `prompt_cache_key` | ✅ sference extension — see [Prompt caching](https://sference.com/docs/guides/prompt-caching) | | `service_tier` | ✅ `"flex"` selects flex; other accepted values serve as default | | `max_queue_wait_s` | ✅ sference extension — see [Max queue wait](https://sference.com/docs/guides/max-queue-wait) | | `n` (multiple completions) | ❌ not supported | | `logprobs`, `top_logprobs` | ❌ not supported | | `logit_bias` | ❌ not supported | ### OpenAI Responses API | Field | Status | | --- | --- | | `model`, `input`, `instructions` | ✅ | | `max_output_tokens`, `temperature`, `top_p` | ✅ | | `tools`, `tool_choice` | ✅ | | `text.format` | ✅ — structured output | | `reasoning`, `include_reasoning`, `enable_thinking` | ✅ | | `background` | ✅ — 24h batch window | | `stream` | ✅ | | `metadata.completion_window` | ✅ `"24h"` (requires `background: true`) | | `metadata.stream_id` | ✅ sference extension — see [Streams](https://sference.com/docs/guides/streams) | | `metadata.completion_webhook`, `metadata.webhook_token` | ✅ — see [Webhooks](https://sference.com/docs/guides/webhooks) | | `prompt_cache_key`, `service_tier`, `max_queue_wait_s` | ✅ | | `store` | ⚠️ accepted; there is no server-side storage | | `previous_response_id`, `conversation` | ❌ ignored — send the conversation each turn | | `include` (web search, file search, computer use) | ❌ ignored — see below | > **No server-side conversation state** > > Every request is answered from the content you send. `previous_response_id` and `conversation` are **removed rather than rejected**, so your request still runs — it just does not see the turn they name. Send the full conversation in `input` on every turn; every OpenAI-compatible SDK does this by default. ### Anthropic Messages API The full field-by-field map, including content blocks, streaming events, and the exact differences from `api.anthropic.com`, is on the [Anthropic Messages API](https://sference.com/docs/api/messages) page. | Field | Status | | --- | --- | | `model`, `max_tokens`, `messages`, `system` | ✅ | | `temperature`, `top_p`, `top_k` | ✅ | | `stop_sequences`, `stream` | ✅ | | `tools`, `tool_choice` | ✅ | | `thinking`, `enable_thinking` | ✅ — `signature` is always empty | | `max_queue_wait_s` | ✅ sference extension | | `cache_control` | ❌ ignored — prefix caching is automatic | | `service_tier`, `background` | ❌ realtime only on this surface | | Server-side tools (web search, code execution, computer use) | ❌ unavailable | ## Not supported anywhere | Feature | Why | | --- | --- | | Server-side tools — web search, file search, code execution, computer use | These are hosted by the provider, not the model. Run them in your own agent loop. | | `n > 1` | One completion per request. Fan out with [concurrent requests](https://sference.com/docs/guides/requests-at-scale) instead. | | Log probabilities | Not exposed by the serving engines we run. | | Fine-tuning endpoints | Custom weights are hosted under an [enterprise agreement](https://sference.com/docs/custom-models), not self-serve. | | Server-side conversation storage | Every request is stateless by design; see below. | ## Statelessness is deliberate sference stores no conversation state. A response is computed from the request body alone, which is what makes the platform auditable: what you sent is what ran, and there is no hidden thread to reason about. It also means the request body is the source of truth for context — pass the whole conversation every turn. ## Checking support programmatically Every model entry on `GET /v1/models` carries its capabilities: ```json { "id": "zai-org/GLM-5.3-Flash", "context_tokens": 1048576, "capabilities": { "image_input": { "supported": true } }, "pricing": { "input_per_million_usd": 0.2, "cached_input_per_million_usd": 0.07, "output_per_million_usd": 0.6 } } ``` Read `capabilities` before routing a request that depends on one — vision in particular. The platform rejects rather than silently drops an unsupported input. - [Support matrix](https://sference.com/docs/support): One-page grid of every endpoint and field. - [Anthropic Messages API](https://sference.com/docs/api/messages): The full Anthropic compatibility surface. - [Models](https://sference.com/docs/models): Catalog ids, capabilities, and pinning. --- # Support matrix > One-page grid: every sference endpoint, what it accepts, and which windows it runs on. Source: https://sference.com/docs/support One page, every surface. Use it to check whether a feature exists before designing around it. ## Endpoints | Endpoint | Method | Window | Notes | | --- | --- | --- | --- | | `/v1/chat/completions` | POST | realtime, flex | OpenAI Chat Completions shape | | `/v1/responses` | POST | realtime, flex, batch | `background: true` for the 24h window | | `/v1/responses/{id}` | GET, DELETE | — | Retrieve or delete a background response | | `/v1/responses/events` | GET | — | Completion feed with a resumable cursor | | `/v1/messages` | POST | realtime | Anthropic Messages shape | | `/v1/embeddings` | POST | realtime | OpenAI embeddings shape | | `/v1/models` | GET | — | Catalog with capabilities and rates | | `/v1/batches` | POST, GET | batch | Create and list batches | | `/v1/batches/{id}` | GET | — | Batch status | | `/v1/batches/{id}/cancel` | POST | — | Cancel a batch | | `/v1/batches/{id}/results` | GET | — | Results as JSON | | `/v1/batches/{id}/results.jsonl` | GET | — | Results as JSONL | | `/v1/files` | POST, GET | — | Upload and list files (batch input) | | `/v1/streams` | POST, GET | batch | Create and list streams | | `/v1/streams/{id}` | GET, PATCH | — | Stream detail and lifecycle | | `/control/v1/usage/*` | GET | — | Usage summaries and breakdowns | | `/control/v1/activity/*` | GET | — | Per-request log and CSV export | | `/control/v1/billing/*` | GET | — | Balance, summary, transactions | Full schemas for every operation are in the [API Reference](https://sference.com/docs/api-reference). ## Windows by surface | Surface | realtime | flex | batch (24h) | | --- | --- | --- | --- | | `/v1/chat/completions` | ✅ | ✅ `service_tier: "flex"` | ❌ | | `/v1/responses` | ✅ | ✅ | ✅ `background: true` | | `/v1/messages` | ✅ | ❌ | ❌ | | `/v1/embeddings` | ✅ | ❌ | ❌ | | Batches | — | ✅ stacks with batch | ✅ | | Streams | — | ✅ stacks with batch | ✅ | Flex and batch stack: a batch submitted with `service_tier: "flex"` bills at `0.7 × 0.5 = 0.35 ×` list. See [Pricing](https://sference.com/docs/pricing). ## Request fields | Field | Chat | Responses | Messages | | --- | --- | --- | --- | | `temperature`, `top_p` | ✅ | ✅ | ✅ | | `top_k` | ✅ | — | ✅ | | `max_tokens` / `max_output_tokens` | ✅ | ✅ | ✅ | | `tools`, `tool_choice` | ✅ | ✅ | ✅ | | Structured output | ✅ `response_format` | ✅ `text.format` | ❌ | | Images | ✅ | ✅ | ✅ | | `stream` | ✅ | ✅ | ✅ | | Reasoning control | ✅ `reasoning_effort`, `enable_thinking` | ✅ `reasoning` | ✅ `thinking` | | `prompt_cache_key` | ✅ | ✅ | ❌ | | `service_tier: "flex"` | ✅ | ✅ | ❌ | | `max_queue_wait_s` | ✅ | ✅ | ✅ | | `background` | ❌ | ✅ | ❌ | | `stream_id` (metadata) | ❌ | ✅ | ❌ | | Webhook (metadata) | ✅ | ✅ | ❌ | | `n > 1` | ❌ | ❌ | ❌ | | Log probabilities | ❌ | ❌ | ❌ | | Server-side tools | ❌ | ❌ | ❌ | | Server-side conversation state | ❌ | ❌ | ❌ | ## Model capabilities | Model | Vision | Context | Input / output per 1M | | --- | --- | ---: | --- | | `moonshotai/Kimi-K3` | ✅ | 1,048,576 | 3.00 / 15.00 | | `zai-org/GLM-5.3` | ❌ | 1,048,576 | 1.20 / 4.20 | | `zai-org/GLM-5.2` | ❌ | 1,048,576 | 1.20 / 4.20 | | `zai-org/GLM-5.3-Flash` | ✅ | 1,048,576 | 0.20 / 0.60 | | `deepseek-ai/DeepSeek-V4.1-Flash` | ✅ | 1,048,576 | 0.50 / 1.50 | | `deepseek-ai/DeepSeek-V4-Flash` | ❌ | 1,048,576 | 0.28 / 0.56 | | `Cloudflare/clef` | ✅ | 65,536 | 0.24 / 0.00 | > **Always read the live catalog** > > The table above is a snapshot. `GET https://api.sference.com/v1/models` is authoritative and reflects your entitlements, including custom models. ## Limits | Limit | Value | | --- | --- | | `max_queue_wait_s` | 5–300 seconds | | Completion window | `24h` (only supported value) | | Streaming admission ceiling | 60s projected TTFT by default | | Sync admission ceiling | 300s projected TTFT by default | | Concurrency | bounded by your account's rate limits; see `GET /v1/models` and your account settings | - [API compatibility](https://sference.com/docs/compatibility): Which fields are honored, and which are ignored. - [API reference](https://sference.com/docs/api-reference): Full schemas for every operation. - [Models](https://sference.com/docs/models): The live catalog and how to read it. --- # Custom Models > Bring your own weights: fine-tunes and distilled open models run on the same API, infrastructure, and audit trail as the catalog. Source: https://sference.com/docs/custom-models Upload your own **Hugging Face** weights and serve them through the same OpenAI-compatible API as the catalog. Fine-tunes and distilled open-weight models run on the same infrastructure, pinned versions, and audit trail as catalog models. **vLLM/SGLang models work out of the box**, and we stand up custom runtimes for models that need them. Custom model hosting is arranged per customer as part of an enterprise agreement. [Talk to us](mailto:hello@sference.com) with the model and your expected volume. > **Requested, then provisioned** > > Custom models are requested from the [console](https://app.sference.com) and provisioned by Sference. A model moves from **pending** to **available** once it is ready to serve; you call it by the same model id you registered. ## Register a model 1. In the [console](https://app.sference.com), open **Custom Models** and choose **Add model**. 2. Enter the **Hugging Face model id** (for example `your-org/your-finetune`) and its **architecture** (the base family it derives from, e.g. Qwen or Llama). 3. If the repository is **private**, mark it as such so we can arrange access. 4. Submit. The model appears with status **pending** while we provision it, then flips to **available**. > **Access to private repos** > > For a private Hugging Face repository, grant Sference read access (or share a scoped token) so the weights can be pulled. Public repositories need no extra step. ## Call your model Once the model is **available**, use it exactly like a catalog model: same endpoints, same request shape, same [pinned-version](https://sference.com/docs/models) guarantees. Reference it by the model id you registered: ```bash sference responses create \ --model "your-org/your-finetune" \ --content "Summarize this contract in three bullet points." \ --wait ``` You can also open it in the **playground** from the console, or route production traffic to it through a dedicated [deployment](https://sference.com/docs/deployments). ## Billing Hosting and per-token rates for custom models are agreed per customer as part of an enterprise agreement, not published as a rate card. Pricing depends on the model, the hardware it needs, and your volume. Every request carries the same request-level logging and audit trail as catalog traffic. ## Next steps - [Deployments](https://sference.com/docs/deployments): Give a custom model dedicated capacity with a performance tier. - [Models](https://sference.com/docs/models): How the catalog, pinned versions, and size classes work. - [Quickstart](https://sference.com/docs/quickstart): API key, CLI install, and your first request. --- # Deployments > Dedicated capacity for a base or custom model: pick a performance tier and Sference provisions and runs it for you. Source: https://sference.com/docs/deployments A **deployment** gives a model **dedicated capacity**: reserved throughput and predictable performance on a private endpoint, separate from the shared serverless pool. Deploy any **catalog** model or one of your own [custom models](https://sference.com/docs/custom-models), with the same OpenAI-compatible API, same pinned versions, same audit trail. ## Performance tiers Instead of choosing hardware, you choose a **performance tier**. Sference maps the tier to the right capacity and provisions it for you, so you describe the outcome you need, not the GPUs behind it. | Tier | Best for | | --- | --- | | **Cost saving** | Lowest cost, for background and latency-tolerant workloads. | | **Standard** | Balanced cost and speed. The default. | | **Turbo** | Highest throughput and lowest latency. | > **Hardware is handled for you** > > You never pick or manage GPUs. Sference selects the hardware behind your chosen tier, provisions it, and keeps it running. ## Request a deployment 1. In the [console](https://app.sference.com), open **Deployments** and choose **Create deployment**. 2. Select the **model**: a base catalog model or one of your **available** custom models. 3. Choose a **region**, a **performance tier**, and the **replica** range (min/max), then give the deployment a name. 4. Submit. Sference provisions and activates it; the status moves from **pending** through **deploying** to **available**. ## Use it Once the deployment is **available**, send traffic to it the same way you call any model on the API: streaming, tool calling, and structured output all work unchanged. Because the model id and request shape are identical to serverless, moving a hot path onto dedicated capacity needs no code changes beyond routing. ## Pricing Dedicated capacity is arranged with your account. **[Talk to us](mailto:hello@sference.com)** to set up a plan that fits your throughput and commitment. ## Next steps - [Custom Models](https://sference.com/docs/custom-models): Bring your own weights, then deploy them. - [Models](https://sference.com/docs/models): Catalog, pinned versions, and size classes. --- # Build with agent frameworks > Build agent loops on sference with LangChain, the OpenAI Agents SDK, and the Anthropic Agent SDK — and pick the window each step runs on. Source: https://sference.com/docs/agents An agent loop is many model calls in sequence: plan, call a tool, read the result, decide again. sference is built for that shape. The turn a user is waiting on runs **realtime**; the steps behind it can run on **flex** or the **24h batch window** — same API, same fleet, same audit trail, one bill. This section shows how to point three popular agent frameworks at sference, how to choose a window per step, and the two patterns every agent needs in production: **structured output** and **sending requests at scale**. | Framework | Page | Why you would pick it | | --- | --- | --- | | **LangChain / LangGraph** | [LangChain](https://sference.com/docs/agents/langchain) | You already build graphs, chains, or RAG pipelines and want sference as the model provider | | **OpenAI Agents SDK** | [OpenAI Agents SDK](https://sference.com/docs/agents/openai-agents-sdk) | Handoffs, guardrails, and tracing; OpenAI-native agent primitives | | **Anthropic Agent SDK** | [Anthropic Agent SDK](https://sference.com/docs/agents/anthropic-agent-sdk) | You use the Claude Agent SDK or the `anthropic` SDK's tool runner and want open models behind it | ## One base URL, three SDKs Every framework below reaches sference the same way: an OpenAI-compatible or Anthropic-compatible base URL plus a `sk_...` key. Nothing else about the framework changes. ```python import os BASE_URL = "https://api.sference.com/v1" API_KEY = os.environ["SFERENCE_API_KEY"] ``` | Framework | Surface it uses | Base URL | | --- | --- | --- | | LangChain (`ChatOpenAI`) | `/v1/chat/completions` | `https://api.sference.com/v1` | | OpenAI Agents SDK | `/v1/responses` | `https://api.sference.com/v1` | | Anthropic Agent SDK | `/v1/messages` | `https://api.sference.com` | > **Model ids are catalog ids** > > There are no `gpt-*` or `claude-*` aliases. Pass a sference catalog id such as `zai-org/GLM-5.3` or `moonshotai/Kimi-K3`. `GET https://api.sference.com/v1/models` lists what your key can call. ## Choosing a window per step The single biggest cost decision in an agent is **which steps need to be synchronous**. Most do not. | Step | Window | How | | --- | --- | --- | | User-facing turn, tool selection the user is waiting on | realtime | default sync call | | Background research, retrieval, summarization | flex | `service_tier: "flex"` | | Long-horizon steps, overnight jobs, evals | batch | `background: true` + `metadata.completion_window: "24h"` | | Bulk fan-out of many independent steps | batch | [Batch API](https://sference.com/docs/guides/batches) | Flex and batch are **0.7×** and **0.5×** list price respectively, and they stack. See [Pricing](https://sference.com/docs/pricing). ```python # The turn the user waits on — realtime answer = client.chat.completions.create(model=MODEL, messages=messages) # A background step nobody is waiting on — flex, 30% cheaper summary = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": doc}], service_tier="flex", ) # An overnight job — batch, 50% cheaper job = client.responses.create( model=MODEL, input=task, background=True, metadata={"completion_window": "24h"}, ) ``` ## Keeping an agent loop cheap and fast Four practices apply to every framework on this page: 1. **Pin the worker with `prompt_cache_key`.** An agent resends the same system prompt and tool definitions on every step. Set `prompt_cache_key` to the run id so those tokens bill at the cached rate instead of full price. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). 2. **Bound the wait with `max_queue_wait_s`.** Fail over rather than hang when the fleet is busy. See [Max queue wait](https://sference.com/docs/guides/max-queue-wait). 3. **Cap `max_output_tokens` on reasoning models.** The cap covers hidden reasoning, so an uncapped thinking step can burn the budget before it acts. 4. **Put the volatile part of the prompt last.** Timestamps, retrieved documents, and the user's question go after the stable system prompt. ## Where to go next - [LangChain](https://sference.com/docs/agents/langchain): ChatOpenAI, LangGraph, and tool-calling agents on sference. - [OpenAI Agents SDK](https://sference.com/docs/agents/openai-agents-sdk): Responses API, handoffs, and background steps. - [Anthropic Agent SDK](https://sference.com/docs/agents/anthropic-agent-sdk): Claude Agent SDK and the anthropic SDK on open models. - [Structured outputs](https://sference.com/docs/guides/structured-outputs): Constrain a step's output to a JSON schema. - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): Fan out concurrent background responses, or submit one batch. - [SferenceCode](https://sference.com/docs/sference-code): Run a coding agent on sference with one command. --- # LangChain > Use sference as the model provider for LangChain and LangGraph agents: ChatOpenAI, tool calling, streaming, and per-step windows. Source: https://sference.com/docs/agents/langchain sference exposes an OpenAI-compatible `/v1/chat/completions`, so LangChain's `ChatOpenAI` works with a base-URL change. Chains, tools, and LangGraph graphs then run on open models with the same code. ## Install ```bash pip install -U langchain-openai ``` ## Point `ChatOpenAI` at sference ```python import os from langchain_openai import ChatOpenAI llm = ChatOpenAI( model="zai-org/GLM-5.3", base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], temperature=0.2, ) ``` Any LangChain component that takes a chat model now runs on sference — chains, output parsers, retrievers, and LangGraph nodes. ## Tool calling LangChain's tool decorator binds to the OpenAI tool-calling shape, which sference serves natively: ```python from langchain_core.tools import tool @tool def get_weather(city: str) -> str: """Return the current weather for a city.""" return weather_api(city) llm_with_tools = llm.bind_tools([get_weather]) response = llm_with_tools.invoke("What is the weather in Ljubljana?") ``` > **Tool reliability varies by model** > > Open models call tools less consistently than frontier closed models on long loops. `moonshotai/Kimi-K3` and `zai-org/GLM-5.3` are the strongest tool callers in the catalog; validate a tool-heavy workflow on your own prompts before committing to it. ## Structured output Use `with_structured_output` for typed results — it maps to `response_format` with a JSON schema: ```python from pydantic import BaseModel, Field class Ticket(BaseModel): severity: str = Field(description="One of: low, medium, high") product: str summary: str extractor = llm.with_structured_output(Ticket) ticket = extractor.invoke(f"Extract fields from this ticket:\n\n{ticket_text}") ``` See [Structured outputs](https://sference.com/docs/guides/structured-outputs) for schema guidance that improves reliability. ## Streaming ```python for chunk in llm.stream("Write a haiku about GPUs."): print(chunk.content, end="", flush=True) ``` Streaming is realtime by default. For long agent steps where nobody is waiting, prefer a background request — see below. ## A LangGraph agent ```python from typing import Annotated, TypedDict from langgraph.graph import StateGraph, END from langgraph.prebuilt import ToolNode class State(TypedDict): messages: Annotated[list, "add_messages"] def call_model(state: State): return {"messages": [llm_with_tools.invoke(state["messages"])]} graph = StateGraph(State) graph.add_node("agent", call_model) graph.add_node("tools", ToolNode([get_weather])) graph.set_entry_point("agent") graph.add_conditional_edges("agent", should_continue, {"tools": "tools", END: END}) graph.add_edge("tools", "agent") app = graph.compile() result = app.invoke({"messages": [("user", "What is the weather in Ljubljana?")]}) ``` ## Cost control in a graph Two levers matter for a multi-step graph: **Pin the prefix cache.** An agent resends the same system prompt and tool definitions on every node. Set `prompt_cache_key` to the graph run id so those tokens bill at the cached rate: ```python llm = ChatOpenAI( model="zai-org/GLM-5.3", base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], model_kwargs={"prompt_cache_key": run_id}, # stable per graph run ) ``` **Move background nodes to flex.** Nodes whose output the user is not waiting on can run 30% cheaper: ```python background_llm = ChatOpenAI( model="zai-org/GLM-5.3", base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], model_kwargs={"service_tier": "flex"}, ) ``` Use `background_llm` for research, summarization, and retrieval-summarizing nodes, and `llm` for the nodes on the critical path. See [Pricing](https://sference.com/docs/pricing) and [Prompt caching](https://sference.com/docs/guides/prompt-caching). ## Bound the wait Add `max_queue_wait_s` to fail fast under congestion instead of holding a graph node open: ```python llm = ChatOpenAI( ..., model_kwargs={"max_queue_wait_s": 20}, ) ``` See [Max queue wait](https://sference.com/docs/guides/max-queue-wait) for the failover pattern. - [Agent SDKs](https://sference.com/docs/agents): Compare frameworks and per-step window choices. - [Prompt caching](https://sference.com/docs/guides/prompt-caching): The largest lever on an agent's bill. - [Structured outputs](https://sference.com/docs/guides/structured-outputs): Typed results from with_structured_output. --- # OpenAI Agents SDK > Run the OpenAI Agents SDK against sference: Responses API, tools, handoffs, guardrails, and background steps on the 24h window. Source: https://sference.com/docs/agents/openai-agents-sdk The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) is built on the Responses API, which sference serves natively. Point the client at sference and agents, handoffs, and guardrails run on open models. ## Install ```bash pip install -U openai-agents ``` ## Point the SDK at sference ```python import os from agents import set_default_openai_client, set_default_openai_api from openai import AsyncOpenAI client = AsyncOpenAI( base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], ) set_default_openai_client(client) set_default_openai_api("responses") ``` Model names are catalog ids: ```python from agents import Agent, Runner agent = Agent( name="Assistant", model="zai-org/GLM-5.3", instructions="You are a concise assistant. Answer in one paragraph.", ) result = await Runner.run(agent, "Explain prefix caching in one paragraph.") print(result.final_output) ``` ## Tools The SDK's function-tool decorator produces the OpenAI tool schema, which sference accepts unchanged: ```python from agents import function_tool @function_tool def search_docs(query: str) -> str: """Search the internal documentation.""" return docs_index.search(query) agent = Agent( name="Docs assistant", model="moonshotai/Kimi-K3", instructions="Answer using search_docs. Cite the document title.", tools=[search_docs], ) result = await Runner.run(agent, "How do I rotate an API key?") ``` ## Structured output Use `output_type` with a Pydantic model — the SDK maps it to a JSON schema on the Responses API: ```python from pydantic import BaseModel class Triage(BaseModel): severity: str team: str summary: str agent = Agent( name="Triage", model="zai-org/GLM-5.3", instructions="Classify the ticket.", output_type=Triage, ) result = await Runner.run(agent, ticket_text) triage: Triage = result.final_output ``` ## Handoffs ```python from agents import Agent, handoff billing = Agent(name="Billing", model="zai-org/GLM-5.3-Flash", instructions="Handle billing questions.") technical = Agent(name="Technical", model="zai-org/GLM-5.3", instructions="Handle technical questions.") triage = Agent( name="Triage", model="zai-org/GLM-5.3-Flash", instructions="Route the user to the right specialist.", handoffs=[handoff(billing), handoff(technical)], ) result = await Runner.run(triage, "My invoice is wrong.") ``` Use a cheap, fast model for the router and a stronger one for the specialist — the catalog spans `GLM-5.3-Flash` (0.20 / 0.60) to `Kimi-K3` (3.00 / 15.00), so routing decisions are the natural place to economize. ## Guardrails ```python from agents import Agent, GuardrailFunctionOutput, input_guardrail, Runner @input_guardrail async def no_pii(ctx, agent, input): flagged = contains_pii(str(input)) return GuardrailFunctionOutput(output_info={"pii": flagged}, tripwire_triggered=flagged) agent = Agent( name="Support", model="zai-org/GLM-5.3", instructions="Answer support questions.", input_guardrails=[no_pii], ) ``` ## Background steps on the 24h window The SDK's `Runner` is synchronous from your code's perspective, but individual steps do not have to be. For long-horizon work — research passes, bulk classification, overnight evaluation — submit the step as a background response and collect it later: ```python response = await client.responses.create( model="zai-org/GLM-5.3", input=research_question, background=True, metadata={"completion_window": "24h"}, ) print(response.id) # collect later, or register a webhook ``` Background responses bill at **0.5 ×** the list rate. For steps where someone is waiting but which are not on the critical path, `service_tier: "flex"` gives **0.7 ×**. See [Pricing](https://sference.com/docs/pricing). > **`store` and conversation state** > > sference answers every request from the content you send — there is no server-side thread to point at. Pass the full conversation in `input` on each turn (the SDK does this by default) and do not rely on `previous_response_id` or a stored `conversation`. See [API compatibility](https://sference.com/docs/compatibility). ## Tracing The SDK's tracing exports to OpenAI's platform by default. To disable it or point it elsewhere: ```python from agents import set_tracing_disabled set_tracing_disabled(True) ``` ## Cost control ```python client = AsyncOpenAI( base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], default_headers={"X-Sference-Run": run_id}, # your own correlation header ) ``` - **`prompt_cache_key`** — set it to the run id so the system prompt and tool definitions bill at the cached rate on every step. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **`max_queue_wait_s`** — bound the wait and fail over. See [Max queue wait](https://sference.com/docs/guides/max-queue-wait). - **Cheap routers, strong specialists** — handoffs and guardrails rarely need the largest model. - [Agent SDKs](https://sference.com/docs/agents): Compare frameworks and per-step window choices. - [API compatibility](https://sference.com/docs/compatibility): Which Responses fields sference honors, and which it ignores. - [Webhooks](https://sference.com/docs/guides/webhooks): Get notified instead of polling background responses. --- # Anthropic Agent SDK > Run the Claude Agent SDK and the anthropic SDK's tool runner against sference's /v1/messages surface, on open models. Source: https://sference.com/docs/agents/anthropic-agent-sdk sference serves the Anthropic Messages API at `POST /v1/messages`, so the `anthropic` Python and TypeScript SDKs — and the Claude Agent SDK built on them — work with a base-URL change. Your agent loop then runs on open models with the same code. ## Install ```bash pip install -U anthropic ``` ## Point the client at sference ```python import os from anthropic import Anthropic client = Anthropic( base_url="https://api.sference.com", api_key=os.environ["SFERENCE_API_KEY"], # sent as x-api-key ) ``` `Authorization: Bearer` works too — pass `auth_token` instead of `api_key` if you prefer. ```python message = client.messages.create( model="moonshotai/Kimi-K3", max_tokens=1024, messages=[{"role": "user", "content": "Explain prefix caching."}], ) print(message.content[0].text) ``` Model names are sference catalog ids. There are no `claude-*` aliases — a Claude model name returns `400`. ## Tool use Anthropic's tool schema is served natively: ```python tools = [{ "name": "get_weather", "description": "Return the current weather for a city.", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }] message = client.messages.create( model="zai-org/GLM-5.3", max_tokens=1024, tools=tools, messages=[{"role": "user", "content": "What is the weather in Ljubljana?"}], ) for block in message.content: if block.type == "tool_use": result = get_weather(**block.input) # append the assistant turn and the tool_result, then call again ``` ## Extended thinking Reasoning models stream `thinking` blocks that clients render normally, but `signature` is always empty — multi-turn thinking continuity is best-effort. See [Extended thinking](https://sference.com/docs/api/messages#extended-thinking). ```python message = client.messages.create( model="moonshotai/Kimi-K3", max_tokens=4096, thinking={"type": "enabled", "budget_tokens": 2048}, messages=[{"role": "user", "content": "Plan a migration."}], ) ``` ## Streaming ```python with client.messages.stream( model="zai-org/GLM-5.3", max_tokens=1024, messages=[{"role": "user", "content": "Write a haiku about GPUs."}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` The SSE event sequence matches `api.anthropic.com`; the full list is on the [Anthropic Messages API](https://sference.com/docs/api/messages) page. ## The Claude Agent SDK The Claude Agent SDK drives Claude Code's agent loop. Point it at sference and the loop runs on open models: ```bash export ANTHROPIC_BASE_URL="https://api.sference.com" export ANTHROPIC_AUTH_TOKEN="$SFERENCE_API_KEY" export ANTHROPIC_MODEL="moonshotai/Kimi-K3" ``` For the CLI-level equivalent with hybrid routing — sference models in the `/model` picker, real Claude models passing through — use the launcher: ```bash sference launch claude ``` See [Claude Code](https://sference.com/docs/sference-code/claude-code) for proxy mode, model recommendations, and known limits. ## Cost control `/v1/messages` is **realtime only** — flex, background, and batch live on the OpenAI-shaped endpoints. Two levers still apply: - **`prompt_cache_key` is not available on `/v1/messages`.** Prefix caching is automatic; hits appear as `cache_read_input_tokens` in the usage block. Keep the stable part of the prompt first so it can be reused. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **`max_queue_wait_s` is accepted** — bound the wait and fail over to another provider. See [Max queue wait](https://sference.com/docs/guides/max-queue-wait). - **For long background steps, use the OpenAI-shaped surfaces.** A research or summarization step that nobody waits on can run on `/v1/responses` with `background: true` at 0.5 × list, while the interactive turn stays on `/v1/messages`. ## Known limits | Limit | Effect | | --- | --- | | Empty thinking `signature` | Multi-turn thinking history degrades; nothing to verify | | No `redacted_thinking` | Anthropic's encrypted thinking blocks are never produced | | `budget_tokens`, `thinking: adaptive`, `effort`, `display` | Accepted and ignored; bound output with `max_tokens` | | No prompt-caching controls | `cache_control` blocks are ignored; prefix caching is automatic | | No server-side tools | Web search, code execution, and computer use are Anthropic-hosted and unavailable | | Realtime only | No `service_tier: "flex"`, no background execution on `/v1/messages` | The complete compatibility surface is on the [Anthropic Messages API](https://sference.com/docs/api/messages) page. - [Anthropic Messages API](https://sference.com/docs/api/messages): Every supported field, block type, and streaming event. - [Claude Code](https://sference.com/docs/sference-code/claude-code): One-command launch with hybrid routing. - [Agent SDKs](https://sference.com/docs/agents): Compare frameworks and per-step window choices. --- # API surfaces > sference serves three request shapes — the OpenAI Chat API, the OpenAI Responses API, and the Anthropic Messages API — over one catalog, one key, and one bill. Source: https://sference.com/docs/api sference speaks three request shapes. Pick the one your code already uses; all three reach the same models, the same key, and the same billing surface. | Surface | Endpoint | Shape | Reference | | --- | --- | --- | --- | | **OpenAI Chat API** | `POST /v1/chat/completions` | OpenAI | [OpenAI Chat API](https://sference.com/docs/api/chat-completions) | | **OpenAI Responses API** | `POST /v1/responses` | OpenAI | [OpenAI Responses API](https://sference.com/docs/api/responses) | | **Anthropic Messages API** | `POST /v1/messages` | Anthropic | [Anthropic Messages API](https://sference.com/docs/api/messages) | ## Which one | You have | Use | Why | | --- | --- | --- | | OpenAI-shaped code, most SDKs and frameworks | **OpenAI Chat API** | The widest compatibility; every OpenAI client speaks it. Also takes `service_tier: "flex"` | | A new integration, or you want background execution | **OpenAI Responses API** | Adds `background: true`, streams, and the 24h window | | Anthropic-shaped code, Claude Code, `ChatAnthropic` | **Anthropic Messages API** | Native Anthropic bodies; no translation layer. Realtime only | | Thousands of rows to run offline | **Batches** | One job, one model, results as JSONL on the [24h window](https://sference.com/docs/guides/batches) | All four share one catalog, one API key, and one billing surface. The three request shapes are OpenAI- or Anthropic-compatible, so switching is a base URL and a model id — no code changes. `/v1/messages` is realtime only; see [Not supported](https://sference.com/docs/api/messages#not-supported). ## What each surface supports | Capability | OpenAI Chat API | OpenAI Responses API | Anthropic Messages API | | --- | --- | --- | --- | | Realtime | ✅ | ✅ | ✅ | | Flex (`service_tier: "flex"`) | ✅ | ✅ | ❌ | | Background / 24h window | ❌ | ✅ | ❌ | | Streaming | ✅ | ✅ | ✅ | | Tools | ✅ | ✅ | ✅ | | Structured output | ✅ `response_format` | ✅ `text.format` | ❌ | | Vision | ✅ | ✅ | ✅ | | Reasoning control | ✅ | ✅ | ✅ | | `prompt_cache_key` | ✅ | ✅ | ❌ | | `max_queue_wait_s` | ✅ | ✅ | ✅ | | Streams (`metadata.stream_id`) | ❌ | ✅ | ❌ | | Webhooks | ✅ | ✅ | ❌ | The full field-by-field grid is on [API compatibility](https://sference.com/docs/compatibility), and the single-page endpoint list is on the [Support matrix](https://sference.com/docs/support). ## Base URLs | Surface | Base URL | | --- | --- | | OpenAI-shaped (OpenAI Chat API, OpenAI Responses API) | `https://api.sference.com/v1` | | Anthropic-shaped (Anthropic Messages API) | `https://api.sference.com` | One API key (`sk_...`) authenticates all three. `/v1/messages` accepts `x-api-key` as well as `Authorization: Bearer`; the OpenAI-shaped surfaces use the bearer header. ## Common to all three - **Model ids are catalog ids** — `zai-org/GLM-5.3`, `moonshotai/Kimi-K3`. There are no `gpt-*` or `claude-*` aliases. `GET /v1/models` lists what your key can call. - **The API is stateless.** Every request is answered from the body you send; there is no server-side conversation to point at. Pass the full conversation each turn. - **Prefix caching is automatic.** Keep the stable part of the prompt first; cached tokens bill at the cached-input rate. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **`max_queue_wait_s` bounds the wait** on all three, so you can fail over instead of hanging. See [Max queue wait](https://sference.com/docs/guides/max-queue-wait). - [OpenAI Chat API](https://sference.com/docs/api/chat-completions): The OpenAI-shaped surface most clients already speak. - [OpenAI Responses API](https://sference.com/docs/api/responses): Background execution, streams, and the 24h window. - [Anthropic Messages API](https://sference.com/docs/api/messages): Anthropic-compatible bodies for the anthropic SDKs and Claude Code. - [API compatibility](https://sference.com/docs/compatibility): Every field, what is honored, and what is ignored. --- # OpenAI Chat API > POST /v1/chat/completions on sference: the OpenAI Chat Completions shape, with flex processing, tools, structured output, vision, and streaming. Source: https://sference.com/docs/api/chat-completions `POST https://api.sference.com/v1/chat/completions` is the OpenAI Chat Completions shape, served natively. It is the surface the widest set of clients already speak, so most migrations are a base URL and a model id. ```bash curl https://api.sference.com/v1/chat/completions \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "messages": [{"role": "user", "content": "Say hello in one sentence."}] }' ``` ## Base URL and authentication | | | | --- | --- | | **Endpoint** | `POST https://api.sference.com/v1/chat/completions` | | **Base URL for OpenAI SDKs** | `https://api.sference.com/v1` | | **Auth** | `Authorization: Bearer sk_...` | | **Key format** | sference keys (`sk_...`), minted in the [console](https://app.sference.com) | ```python import os from openai import OpenAI client = OpenAI( base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], ) response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": "Say hello in one sentence."}], ) print(response.choices[0].message.content) ``` > **Model ids are catalog ids** > > There are no `gpt-*` aliases. Pass a sference catalog id such as `zai-org/GLM-5.3` or `moonshotai/Kimi-K3`. `GET /v1/models` lists what your key can call, with capabilities and rates. ## Request fields ### Core | Field | Notes | | --- | --- | | `model` | Catalog id | | `messages` | `system` / `user` / `assistant` / `tool` roles; content may be a string or a list of parts | | `stream` | SSE stream of `chat.completion.chunk` objects | | `stream_options.include_usage` | Emits a final usage chunk — required to read token counts while streaming | | `temperature`, `top_p` | Standard sampling | | `top_k`, `min_p` | Additional sampling controls beyond OpenAI's set | | `presence_penalty`, `frequency_penalty`, `repetition_penalty` | Penalties | | `max_tokens`, `max_completion_tokens` | Output cap. On reasoning models this covers hidden reasoning too | | `stop`, `seed` | Stop sequences and deterministic sampling | ### Tools and structured output | Field | Notes | | --- | --- | | `tools`, `tool_choice` | OpenAI tool-calling shape, served natively | | `response_format` | `{"type": "json_schema", ...}` or `{"type": "json_object"}` — see [Structured outputs](https://sference.com/docs/guides/structured-outputs) | ### Reasoning | Field | Notes | | --- | --- | | `reasoning_effort` | `low` / `medium` / `high`, or an integer, on reasoning models | | `enable_thinking` | Explicit on/off for thinking | ### sference extensions | Field | Notes | | --- | --- | | `service_tier` | `"flex"` selects flex processing at 0.7 × list. Other accepted values serve as default | | `prompt_cache_key` | Stable per-session id; pins the worker so the shared prefix bills at the cached rate — see [Prompt caching](https://sference.com/docs/guides/prompt-caching) | | `max_queue_wait_s` | 5–300s admission bound; fails fast under congestion — see [Max queue wait](https://sference.com/docs/guides/max-queue-wait) | ## Streaming ```python stream = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": "Write a haiku about GPUs."}], stream=True, stream_options={"include_usage": True}, ) for chunk in stream: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` Set `stream_options.include_usage` if you need the token counts — without it the final usage chunk is omitted. ## Tool calling ```python tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Return the current weather for a city.", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": "What is the weather in Ljubljana?"}], tools=tools, ) tool_call = response.choices[0].message.tool_calls[0] # Run the tool, append the assistant turn and a {"role": "tool", ...} message, call again ``` > **Tool reliability varies by model** > > Open models call tools less consistently than frontier closed models on long loops. `moonshotai/Kimi-K3` and `zai-org/GLM-5.3` are the strongest tool callers in the catalog; validate a tool-heavy workflow on your own prompts. ## Vision Send images as content parts when the model supports it: ```python messages=[{ "role": "user", "content": [ {"type": "text", "text": "What is the total on this invoice?"}, {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}, ], }] ``` Check `capabilities.image_input.supported` on `GET /v1/models` first — the platform rejects an image sent to a text-only model. See [Images and vision](https://sference.com/docs/guides/vision). ## Response shape Standard OpenAI Chat Completions: `id`, `object`, `created`, `model`, `choices[]` with `message` and `finish_reason`, and a `usage` block. ```json { "id": "chatcmpl-...", "object": "chat.completion", "model": "zai-org/GLM-5.3", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "Hello!" }, "finish_reason": "stop" }], "usage": { "prompt_tokens": 18, "completion_tokens": 3, "total_tokens": 21, "prompt_tokens_details": { "cached_tokens": 0 } } } ``` `usage.prompt_tokens_details.cached_tokens` is the number of prompt tokens served from the prefix cache. Reasoning models may also return `reasoning_content` on the message. ## Processing modes `/v1/chat/completions` runs **realtime** by default and supports **flex**: ```python response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": doc}], service_tier="flex", # 0.7x list, may queue up to 30 minutes ) ``` Background execution and batches are on the [OpenAI Responses API](https://sference.com/docs/api/responses) surface and the [Batch API](https://sference.com/docs/guides/batches). ## Errors Standard OpenAI error envelope with `error.type`, `error.code`, and `error.message`. Two codes are worth handling explicitly: | Code | Meaning | | --- | --- | | `max_queue_wait_exceeded` | The admission bound was hit — retry against a fallback. See [Max queue wait](https://sference.com/docs/guides/max-queue-wait) | | `408` on a flex request | Flex could not complete in time; the OpenAI SDKs retry automatically | ## Not supported | Field | Why | | --- | --- | | `n` (multiple completions) | One completion per request. Fan out with [concurrent requests](https://sference.com/docs/guides/requests-at-scale) | | `logprobs`, `top_logprobs` | Not exposed by the serving engines | | `logit_bias` | Not supported | The full grid, including what is ignored rather than rejected, is on [API compatibility](https://sference.com/docs/compatibility). - [OpenAI Responses API](https://sference.com/docs/api/responses): Background execution, streams, and the 24h window. - [Messages](https://sference.com/docs/api/messages): The Anthropic-compatible surface. - [Structured outputs](https://sference.com/docs/guides/structured-outputs): response_format with a JSON schema. - [API compatibility](https://sference.com/docs/compatibility): Every field across all three surfaces. --- # OpenAI Responses API > POST /v1/responses on sference: the OpenAI Responses shape with background execution, streams, the 24h window, and the events API. Source: https://sference.com/docs/api/responses `POST https://api.sference.com/v1/responses` is the OpenAI Responses shape, and it is the surface with the most sference-specific capability: **background execution**, the **24h completion window**, and the **events API**. Inference is modeled as a **response** resource: you create it, observe status, and read structured **output** (text, optional reasoning, errors). A response is either **realtime** (blocking, returns when inference finishes) or **background** (`background: true`, polled or streamed to completion). ```bash curl https://api.sference.com/v1/responses \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "input": "Summarize the case for prefix caching." }' ``` > **Where to read schemas** > > Open **API Reference** in the sidebar → **Inference** for `POST/GET /v1/responses` and `/v1/responses/events`. Everything is generated from the same OpenAPI document as production. ## Mental model 1. **`POST /v1/responses`** — create the work (`model`, `input`, optional sampling knobs, `metadata`). 2. **`GET /v1/responses/{id}`** — poll `status`, `output`, and `error` until terminal. 3. **`GET /v1/responses/events`** — cursor through completion events instead of re-reading the full document. 4. **Streams** — group related responses under one `stream_id` with a cost rollup and a resumable cursor. See [Streams](https://sference.com/docs/guides/streams). ## Request fields | Field | Notes | | --- | --- | | `model` | Catalog id | | `input` | A string (shorthand for one user message) or a list of input items | | `instructions` | Developer/system instructions | | `max_output_tokens` | Output cap. On reasoning models this covers hidden reasoning too | | `temperature`, `top_p` | Sampling | | `tools`, `tool_choice` | OpenAI tool shape | | `text.format` | Structured output — see [Structured outputs](https://sference.com/docs/guides/structured-outputs) | | `reasoning`, `include_reasoning`, `enable_thinking` | Reasoning controls | | `include` | Accepted; server-side tools (web search, file search, computer use) are not available | | `stream` | SSE stream | | `background` | Run on the 24h window and return immediately | | `metadata.completion_window` | `"24h"` — the only supported value; requires `background: true` | | `metadata.stream_id` | Attach the response to a [stream](https://sference.com/docs/guides/streams) | | `metadata.completion_webhook`, `metadata.webhook_token` | Callback delivery — see [Webhooks](https://sference.com/docs/guides/webhooks) | | `prompt_cache_key` | Pins the worker for prefix-cache hits — see [Prompt caching](https://sference.com/docs/guides/prompt-caching) | | `service_tier` | `"flex"` for 0.7 × list | | `max_queue_wait_s` | 5–300s admission bound | ## Background execution Set `background: true` and the call returns a response id immediately instead of blocking. Collect it later by polling or by webhook. ```python response = client.responses.create( model="zai-org/GLM-5.3", input="Summarize this 200-page filing.", background=True, metadata={"completion_window": "24h"}, ) print(response.id) # collect later ``` ```python # Later r = client.responses.retrieve(response.id) if r.status == "completed": print(r.output_text) ``` Background responses bill at **0.5 ×** list. This is the surface to use for long-horizon agent steps, overnight jobs, and anything nobody is waiting on. > **Collect on the finish reason, not on status alone** > > On a reasoning model, `max_output_tokens` covers hidden reasoning as well as the answer. A response can settle `incomplete` with a perfectly good answer attached. Check the incomplete detail and use the output if it is there rather than discarding it. ## Streaming ```python with client.responses.stream( model="zai-org/GLM-5.3", input="Write a haiku about GPUs.", ) as stream: for event in stream: if event.type == "response.output_text.delta": print(event.delta, end="", flush=True) ``` ## The events API `GET /v1/responses/events` returns completions in order with a cursor, so a consumer can resume where it stopped. Pass `stream_id` for one stream's feed, or omit it for the account-wide feed of every background response. | Parameter | Meaning | | --- | --- | | `stream_id` | One stream's events; omit for the account-wide feed | | `limit` | Page size | | `starting_after` / `ending_before` | Cursor by completion id | | `wait_ms` | Long-poll: hold the request open for new events | ```python from sference_sdk import SferenceClient client = SferenceClient() for event in client.iter_responses_events(consumer_name="worker-1"): process(event) ``` The SDK's checkpoint helpers persist the cursor per consumer, so a restarted worker resumes rather than replaying. See [Streams](https://sference.com/docs/guides/streams). ## Processing modes | Mode | How | Price | | --- | --- | --- | | **realtime** | blocking `POST /v1/responses` (no `background`) | list | | **flex** | `service_tier: "flex"` | 0.7 × list | | **batch** | `background: true` + `metadata.completion_window: "24h"` | 0.5 × list | All three share the same models and request shape; only scheduling priority and price differ. Responses echo the tier actually served in `service_tier`. For large static backfills prefer the [Batch API](https://sference.com/docs/guides/batches) — cheaper, simpler to retry, easier to reconcile. ## Statelessness Every request is answered from the content you send. `previous_response_id` and `conversation` are **removed rather than rejected**, so your request still runs — it just does not see the turn they name. Send the full conversation in `input` on every turn; every OpenAI-compatible SDK does this by default. See [API compatibility](https://sference.com/docs/compatibility). - [OpenAI Chat API](https://sference.com/docs/api/chat-completions): The OpenAI surface most clients already speak. - [Messages](https://sference.com/docs/api/messages): The Anthropic-compatible surface. - [Streams](https://sference.com/docs/guides/streams): Group responses, roll up cost, consume with a cursor. - [Batch API](https://sference.com/docs/guides/batches): JSONL jobs on the same 24h window. --- # Anthropic Messages API > POST /v1/messages on sference: an Anthropic Messages-compatible endpoint for open models. Auth, streaming SSE, tool use, extended thinking, vision, usage, and the exact compatibility surface. Source: https://sference.com/docs/api/messages `POST https://api.sference.com/v1/messages` is an **Anthropic Messages-compatible** endpoint. Anything that already speaks the [Anthropic Messages API](https://platform.claude.com/docs/en/api/messages) points at sference by changing the base URL and the model id. The `anthropic` Python/TypeScript SDKs, [Claude Code](https://sference.com/docs/sference-code/claude-code), LangChain's `ChatAnthropic`, and your own client all point at sference by changing the base URL and the model id. The request and response bodies are Anthropic-shaped. The models behind them are open-weight checkpoints from the [sference catalog](https://sference.com/docs/models), with the same logging and audit trail as every other endpoint. Choosing between the surfaces? See [API surfaces](https://sference.com/docs/api) for the comparison. `/v1/messages` is **realtime only**; see [Not supported](#not-supported). ## Base URL and authentication | | | | --- | --- | | **Endpoint** | `POST https://api.sference.com/v1/messages` | | **Base URL for Anthropic SDKs** | `https://api.sference.com` | | **Auth** | `x-api-key: sk_...` **or** `Authorization: Bearer sk_...` | | **Key format** | sference keys (`sk_...`), minted in the [console](https://app.sference.com) | Both header styles work, so an Anthropic SDK configured with `api_key` (which sends `x-api-key`) and a client configured with a bearer token both authenticate unchanged. `anthropic-version` is accepted and ignored; there are no dated API versions on sference. A `?beta=true` query parameter is accepted and ignored; Claude Code sends it on every request. ## Your first request ```bash curl https://api.sference.com/v1/messages \ -H "x-api-key: $SFERENCE_API_KEY" \ -H "content-type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "max_tokens": 1024, "messages": [ { "role": "user", "content": "Say hello in one sentence." } ] }' ``` ```json { "id": "msg_0f8a1c2e-...", "type": "message", "role": "assistant", "model": "zai-org/GLM-5.3", "content": [{ "type": "text", "text": "Hello, good to meet you." }], "stop_reason": "end_turn", "usage": { "input_tokens": 14, "output_tokens": 9 } } ``` ### Anthropic Python SDK ```python from anthropic import Anthropic client = Anthropic( base_url="https://api.sference.com", api_key="sk_...", # your sference key, not an Anthropic key ) message = client.messages.create( model="zai-org/GLM-5.3", max_tokens=1024, messages=[{"role": "user", "content": "Say hello in one sentence."}], ) print(message.content[0].text) ``` ### Anthropic TypeScript SDK ```ts import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic({ baseURL: "https://api.sference.com", apiKey: process.env.SFERENCE_API_KEY, }); const message = await client.messages.create({ model: "zai-org/GLM-5.3", max_tokens: 1024, messages: [{ role: "user", content: "Say hello in one sentence." }], }); ``` > **Model ids are sference catalog ids** > > Pass a catalog id such as `zai-org/GLM-5.3` or `moonshotai/Kimi-K3`, **not** a Claude model name. There are no `claude-*` aliases; `claude-sonnet-4-5` returns **400**. Matching is case-insensitive, but the id must exist in your catalog. List what your key can reach with `GET /v1/models`, and see [Models](https://sference.com/docs/models) for how to choose. ## Request fields ### Supported | Field | Notes | | --- | --- | | `model` | **Required.** sference catalog id. | | `max_tokens` | **Required**, as in the Anthropic API. | | `messages[]` | **Required.** `role` is `user` or `assistant`; `content` is a string or a block array. | | `system` | String or `text` block array. Folded into one leading system message. | | `stream` | `true` emits Anthropic SSE; see [Streaming](#streaming). | | `temperature` | Forwarded to the engine. | | `top_p` | Forwarded to the engine. | | `top_k` | Forwarded to the engine (Anthropic-only knob; it has no OpenAI equivalent). | | `stop_sequences[]` | Forwarded as the engine's stop strings. | | `tools[]` | Anthropic-native (`name` + `input_schema`) **and** OpenAI-shaped (`type: "function"`) definitions both accepted. | | `tool_choice` | `auto`, `none`, and `required` are honored (string or object form), as is OpenAI's forced-function object. Defaults to `auto` when `tools` are present. See the caveat under [Tool use](#tool-use). | | `thinking` | `{"type": "enabled"}` / `{"type": "disabled"}`; see [Extended thinking](#extended-thinking). | | `enable_thinking` | sference extension; a plain boolean, and it wins over `thinking`. | ### Content blocks | Block | Direction | Support | | --- | --- | --- | | `text` | in / out | Full. | | `tool_use` | in / out | Full; see [Tool use](#tool-use). | | `tool_result` | in | Full, including `content` block arrays. | | `thinking` | in / out | Text is preserved; `signature` is always `""`. | | `image` | in | On vision models only; see [Images](#images). | | `document`, `redacted_thinking`, `search_result`, server tool blocks | in | **Rejected with 400** (`Unsupported content block type`). | Unknown **top-level** fields are ignored rather than rejected, so a client that sends `metadata`, `service_tier`, or `container` gets a normal response; those values simply have no effect. Unknown **content block types** are a hard 400, because silently dropping message content would change what the model sees. ## Streaming Set `stream: true` to receive Server-Sent Events in Anthropic's wire format. Tokens are streamed as the engine produces them. ```bash curl -N https://api.sference.com/v1/messages \ -H "x-api-key: $SFERENCE_API_KEY" \ -H "content-type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "max_tokens": 1024, "stream": true, "messages": [{ "role": "user", "content": "Count to five." }] }' ``` Events emitted, in order: | Event | Notes | | --- | --- | | `message_start` | `usage.input_tokens` is populated once the prompt is tokenized; the cache counters in this frame are placeholders. | | `content_block_start` | For each `thinking`, `text`, and `tool_use` block. | | `content_block_delta` | `thinking_delta`, `signature_delta` (always empty), `text_delta`, and `input_json_delta` for tool arguments. | | `content_block_stop` | Per block. | | `message_delta` | Carries the final `stop_reason` and the authoritative `usage`, including `cache_read_input_tokens`. | | `message_stop` | Terminal frame. | | `ping` | Keepalive while the request is queued or generating, so intermediaries do not reap a healthy connection during a long time-to-first-token. | | `error` | Terminal. `{"type": "error", "error": {"type": "api_error", "message": "..."}}`. | Two details worth coding against: - **Tool arguments arrive as one `input_json_delta`** containing the complete JSON, not as incremental fragments. Clients that concatenate partials work fine; clients that try to parse each fragment as standalone JSON also work. - **`cache_read_input_tokens` only appears on `message_delta`.** The prefix-cache hit is not known when `message_start` is written. If a stream ends without a final frame, whether a timeout or a worker that dropped, the endpoint emits an `error` event before closing, so a client is never left hanging on an open connection. ## Tool use Anthropic-native tool definitions work as written: ```json { "model": "moonshotai/Kimi-K3", "max_tokens": 1024, "messages": [{ "role": "user", "content": "What files are in the repo root?" }], "tools": [ { "name": "list_files", "description": "List files in a directory", "input_schema": { "type": "object", "properties": { "path": { "type": "string" } }, "required": ["path"] } } ] } ``` The model replies with `stop_reason: "tool_use"` and a `tool_use` block carrying `id`, `name`, and a parsed `input` object. Send the result back as a `tool_result` block in a `user` turn, keyed by `tool_use_id`, exactly as with Anthropic. Notes specific to sference: - **OpenAI-shaped tools are also accepted.** A `tools[]` entry with `type: "function"` and a nested `function.name` / `function.parameters` is normalized alongside the Anthropic `name` / `input_schema` form. Mixed arrays are fine. This exists because some Anthropic-compatible clients send the OpenAI shape. - **`tool_choice` forcing is partial.** `auto`, `none`, and `required` are normalized and honored, in string or object form, as is OpenAI's `{"type": "function", "function": {"name": "..."}}`. Anthropic's `{"type": "any"}` and `{"type": "tool", "name": "..."}` are forwarded to the engine unchanged and are **not** reliably honored; if you must force a specific tool, send the OpenAI object form. - **Malformed tool arguments are repaired, not dropped.** If a model emits arguments that are not valid JSON, sference retries a repair pass and, failing that, hands you `{"__malformed_arguments__": ""}` rather than an empty `input`. You always see what the model actually produced. - **Tool quality tracks the model.** Tool calling on open models is parser-dependent; `moonshotai/Kimi-K3` and `zai-org/GLM-5.3` are the strongest agentic choices in the catalog today. ## Extended thinking Reasoning models return their chain of thought as a `thinking` block ahead of the `text` block, matching Anthropic's response shape. Thinking is enabled by any of: - `thinking: {"type": "enabled"}` (Anthropic's spelling; `{"type": "disabled"}` turns it off) - `enable_thinking: true` (sference extension, checked first, so it overrides `thinking`) - **nothing at all**, on a model whose family reasons by default (Qwen, DeepSeek-R1, Kimi, MiniMax, GLM, Hy3). Anthropic clients omit `thinking` for third-party models, so sference defaults it on where the model expects it. Models the catalog marks as non-reasoning always run with thinking off, even if you ask for it; forcing a `` block on a model that has none makes the decoder treat the entire reply as reasoning and return empty `content`. > **Signatures are empty** > > `thinking` blocks come back with `signature: ""`. Anthropic's cryptographic thinking signatures are not implemented, and `redacted_thinking` blocks are not produced. sference re-injects inbound thinking text on multi-turn requests, but a client that validates signatures will not find a valid one. > > `budget_tokens`, `thinking: {"type": "adaptive"}`, `effort`, and `display` are accepted and ignored; use `max_tokens` to bound total output. ## Images Image blocks are honored on vision-capable catalog models (`moonshotai/Kimi-K3`, `deepseek-ai/DeepSeek-V4.1-Flash` and `zai-org/GLM-5.3-Flash` today). Both `base64` and `url` sources work, in user turns and inside `tool_result` content. On a **text-only** model, sference does not fail the request. Image blocks are replaced with `[image omitted: this model has no vision]`, a system notice tells the model it has no vision, and a user-turn reminder discourages retrying screenshot tools. This keeps agent loops, which happily attach screenshots, from spinning on a capability the model does not have. If you need vision, pick a vision model; if you get the sentinel, that is why. ## Response shape ```json { "id": "msg_", "type": "message", "role": "assistant", "model": "zai-org/GLM-5.3", "content": [ { "type": "thinking", "thinking": "...", "signature": "" }, { "type": "text", "text": "..." }, { "type": "tool_use", "id": "toolu_...", "name": "list_files", "input": { "path": "." } } ], "stop_reason": "end_turn", "usage": { "input_tokens": 812, "output_tokens": 143, "cache_read_input_tokens": 640 } } ``` - **`id`** is `msg_` plus the sference request id, the same id you look up in the console and in `GET /control/v1/activity/{request_id}`. That makes every Anthropic-shaped call traceable in the audit trail without extra bookkeeping. - **`stop_reason`** is `tool_use` when the turn ends in a tool call, `max_tokens` when the completion was truncated, otherwise `end_turn`. - **`usage.cache_read_input_tokens`** appears when the prefix cache served part of the prompt; cached input tokens are billed at the lower cached rate. There is no `cache_creation_input_tokens`; caching is automatic and never billed as a write. - **`stop_sequence`** is not returned on non-streaming responses (streaming `message_delta` carries `stop_sequence: null`). ## Errors HTTP status codes follow the usual semantics, but **error bodies are sference-shaped, not Anthropic-shaped**: ```json { "status_code": 400, "detail": "Unsupported content block type: 'document'", "extra": {} } ``` Anthropic SDKs still raise the right exception class from the status code, but `error.type` / `error.message` will not be populated the way they are against api.anthropic.com. Read `detail`. (SSE `error` **events** during a stream *are* Anthropic-shaped.) | Status | Cause | | --- | --- | | **400** | Unknown model, unsupported content block, malformed body, image that failed validation or fetch. | | **401** | Missing or revoked key. | | **402** | Team balance exhausted (no negative balance allowance). | | **504** | Inference did not complete within the sync wait (600s default). The request is cancelled server-side, so no work is billed after the timeout. | ## Not supported `/v1/messages` is a **realtime** endpoint. Anything queued, discounted, or bulk lives on the other surfaces: | Not on `/v1/messages` | Use instead | | --- | --- | | `service_tier: "flex"` (ignored here) | `/v1/chat/completions` or `/v1/responses`; see [processing modes](https://sference.com/docs/api/responses#processing-modes) | | Background / batch execution | `POST /v1/responses` with `background: true` | | Bulk jobs on the 24h window | [`POST /v1/batches`](https://sference.com/docs/guides/batches) | | `POST /v1/messages/count_tokens` | Not implemented; read `usage.input_tokens` off a real response | | Batches, Files, Models list in Anthropic's shape | sference-native `/v1/batches`, `/v1/models` | | Prompt caching **controls** (`cache_control` blocks, `anthropic-beta` headers) | Prefix caching is automatic; hits are reported as `cache_read_input_tokens` | | MCP connectors, web search, code execution, computer use | Not implemented; server-side tools are Anthropic-hosted | | Citations, `document` blocks, PDF input | Not implemented | | Thinking signatures, `redacted_thinking` | Not implemented; see [Extended thinking](#extended-thinking) | ## Differences from api.anthropic.com at a glance | | Anthropic | sference `/v1/messages` | | --- | --- | --- | | Models | `claude-*` | sference catalog ids (open weights, [pinned](https://sference.com/docs/models)) | | Auth | `x-api-key` | `x-api-key` **or** `Authorization: Bearer` | | Versioning | `anthropic-version` required | Accepted and ignored | | Unknown top-level fields | Rejected | Ignored | | Error body | `{"type": "error", "error": {...}}` | `{"status_code", "detail", "extra"}` | | Thinking signatures | Cryptographically signed | Always `""` | | Prompt caching | Explicit `cache_control` | Automatic prefix cache, read-side reporting only | | Processing tiers | Standard / batch / priority | Realtime only on this endpoint | ## Next steps - [Claude Code](https://sference.com/docs/sference-code/claude-code): Run Claude Code against open models: one-command launch, hybrid routing, manual config. - [Models](https://sference.com/docs/models): Catalog ids, pinning, and which checkpoints are strongest for agentic tool use. - [API Reference](https://sference.com/docs/api-reference): Generated schemas for /v1/messages and every other operation. - [Quickstart](https://sference.com/docs/quickstart): API keys, the CLI, and your first request on any endpoint. --- # Batch inference > Enqueue thousands of rows on the 24h window: POST /v1/batches, poll status, download JSONL results. Source: https://sference.com/docs/guides/batches **Batch** is the big hammer on sference. Reach for it when volume, not latency, is the constraint: send an array of inference rows and let the scheduler place work on spot-friendly GPUs, completing within the **24h** completion window (`"24h"` is the only supported value). For user-facing inference, use **realtime** sync endpoints (`/v1/chat/completions`, [`/v1/messages`](https://sference.com/docs/api/messages), blocking `/v1/responses`); for discounted sync work that can tolerate queuing, use [`service_tier: "flex"`](https://sference.com/docs/api/responses#processing-modes) on `/v1/chat/completions` or `/v1/responses`. > **OpenAPI is canonical** > > Use **API Reference → Inference → Batch API** for field-level detail on `POST /v1/batches`, cancellation, and the results download routes. ## Typical control loop 1. **`POST /v1/batches`**: send an inline `requests[]` array (each row has optional `custom_id` and a `body` object). 2. **`GET /v1/batches/{batch_id}`**: poll `status`, token totals, and `request_count` while work drains. 3. **`GET /v1/batches/{batch_id}/results`** or **`GET /v1/batches/{batch_id}/results.jsonl`**: fetch per-row outcomes once the batch is terminal (`completed`, `failed`, or `cancelled`). Retries should be **idempotent**: set a stable **`custom_id`** per row when you need correlation across replays. ## One model per batch Every row in `requests[]` must use the **same** `body.model`. Mixing models in one batch returns **HTTP 400** before anything is enqueued: ```text requests[1] (custom_id="row-b"): all batch rows must use the same model; expected "deepseek-ai/DeepSeek-V4-Flash-0731", got "other-model" ``` The model must also be **available in your platform catalog** (`platform_status=available`). Unknown or unavailable ids fail at create with `Model "…" is not available for inference`. Split multi-model workloads into separate batch jobs (or use [OpenAI Responses API](https://sference.com/docs/api/responses) with per-request models). ## Request shape ```json { "window": "24h", "requests": [ { "custom_id": "row-a", "body": { "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{ "role": "user", "content": "Hello" }] } }, { "custom_id": "row-b", "body": { "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{ "role": "user", "content": "Summarize this." }] } } ] } ``` Base URL: **`https://api.sference.com/v1`** (Bearer API key). ## Row `body` shapes (normalize → validate) At create time the API **normalizes** each row `body` to internal chat-completions format, **validates** it, then persists. Invalid rows return **HTTP 400** with `requests[i]` and optional `custom_id`. Nothing is enqueued, so you never get late worker failures like `body.messages must be a list`. ### Chat completions ```json { "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{ "role": "user", "content": "Summarize this." }], "temperature": 0.2, "max_tokens": 512 } ``` `messages` must be a **non-empty** array. Optional fields match `POST /v1/chat/completions` (`tools`, `tool_choice`, …). ### Responses API (normalized at create) Use the same fields as `POST /v1/responses`. The API converts `input` → `messages`, `max_output_tokens` → `max_tokens`, and similar mappings before enqueue: ```json { "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "input": [{ "role": "user", "content": "Summarize this." }], "instructions": "Reply in one sentence.", "max_output_tokens": 512 } ``` String shorthand for `input` is supported. After create, stored rows always contain `messages`; workers never see raw Responses shape. ### Rejected at create | Case | Example error | |------|----------------| | Missing `messages` and `input` | `body must include a non-empty messages list or Responses API input` | | Empty `messages: []` | `body.messages must be a non-empty list` | | Invalid Responses payload | Field-level validation on `input`, etc. | | `background: true` in row body | `background is not supported in batch request bodies` | | Unknown or unavailable model | `requests[i]`: `Model "…" is not available for inference` | | Mixed models across rows | `requests[i]`: `all batch rows must use the same model; expected "…", got "…"` | > **Do not set background on batch rows** > > Batches are already on the 24h window. Use `background: true` on **`POST /v1/responses`** for per-request background jobs, not inside batch row bodies. ## JSONL via CLI / SDK For file-based workflows, use **`sference batch submit`** or **`client.submit_batch(input_file=...)`**. Each JSONL line is one batch row. Use the same model on every line (or pass **`--model`** / `model=` for content-only lines only). **OpenAI-style envelope**: the SDK sends only `custom_id` + inner `body`; `method` / `url` are ignored: ```jsonl {"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"…","messages":[{"role":"user","content":"hi"}]}} {"custom_id":"b","method":"POST","url":"/v1/responses","body":{"model":"…","input":[{"role":"user","content":"hi"}]}} ``` **Content-only**: requires global `model=` on submit: ```jsonl {"content":"Classify this log line."} ``` See **[CLI](https://sference.com/docs/cli)** and the [OSS CLI README](https://github.com/s-ference/sference/blob/main/cli/README.md) for subcommands and **`sference batch stream`**. ## Not OpenAI’s file-upload Batch API sference batches differ from [OpenAI Batch](https://platform.openai.com/docs/guides/batch): - No **`POST /v1/files`** or batch objects referencing uploaded JSONL file IDs. - Create uses **inline `requests[]`**, not a separate file upload step. - Result rows use **`result_json` / `error_json`**, not OpenAI’s batch result envelope. For OpenAI **Responses** workloads at scale, use Responses-shaped **`body`** rows in a batch (above) or **`POST /v1/responses`** with `background: true` per request. ## CLI quick example ```bash sference auth login # or export SFERENCE_API_KEY='sk_...' in CI sference batch submit --input-file ./workload.jsonl --model deepseek-ai/DeepSeek-V4-Flash-0731 --window 24h sference batch wait --batch-id --timeout 86400 sference batch download-results --batch-id --out ./results.jsonl ``` ## Python SDK quick example ```python from sference_sdk import SferenceClient client = SferenceClient(api_key="sk_...") batch = client.submit_batch( input_file="./workload.jsonl", model="deepseek-ai/DeepSeek-V4-Flash-0731", window="24h", ) done = client.wait_for_completion(batch.id, poll_interval=5.0, timeout=86_400.0) client.download_results_jsonl(done.id, out="./results.jsonl") ``` Pass an explicit **`timeout`** to `wait_for_completion`; the default is 30 seconds. ## When *not* to use batches - **Interactive** assistants with typing indicators → realtime [OpenAI Responses API](https://sference.com/docs/api/responses). - **Latency-tolerant sync work** (unattended agents, retries OK) → `service_tier: "flex"` on the sync endpoints is simpler than a batch job; see [Processing modes](https://sference.com/docs/api/responses#processing-modes). ## Cross-links - [Quickstart](https://sference.com/docs/quickstart): Authenticate and hit `/v1` for the first time. - [Models](https://sference.com/docs/models): Match model tier to workload cost. - [CLI](https://sference.com/docs/cli): JSONL submit, wait, and download. - [Python SDK](https://sference.com/docs/sdk): submit_batch and agent prompts. - [API Reference](https://sference.com/docs/api-reference): Batch routes with live schemas. --- # Streams > Group related requests into a named stream: one window, one cost rollup, and a resumable cursor for workers that consume results incrementally. Source: https://sference.com/docs/guides/streams A **stream** is a named grouping of inference requests that share one completion window, one cost rollup, and one lifecycle. It is how you give a workload an identity on the platform instead of a pile of unrelated request ids. Streams are the right primitive when: - A job produces many requests and you want **one cost number** for the job. - Several workers consume results from the same queue and need a **resumable cursor** so a restarted worker does not reprocess everything. - You want to **cancel or archive** a whole workload in one call. ## Create a stream ```bash curl -X POST https://api.sference.com/v1/streams \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "nightly-eval-2026-10-13", "window": "24h"}' ``` ```python from sference_sdk import SferenceClient client = SferenceClient() stream = client.create_stream(name="nightly-eval-2026-10-13", window="24h") ``` `24h` is the only supported window value. ## Attach requests to a stream Include `stream_id` in the response's `metadata`: ```python response = client.create_response( model="zai-org/GLM-5.3", input=prompt, background=True, metadata={"stream_id": stream.id, "completion_window": "24h"}, ) ``` Every request carrying that `stream_id` belongs to the stream. Requests may be submitted from any number of processes or machines. ## Read the rollup `GET /v1/streams/{stream_id}` returns live counters: ```json { "id": "0199c3e2-...", "name": "nightly-eval-2026-10-13", "window": "24h", "status": "open", "total_items": 12000, "pending_items": 340, "running_items": 64, "completed_items": 11580, "failed_items": 16, "cancelled_items": 0, "completion_ratio": 0.965, "total_prompt_tokens": 48210000, "total_completion_tokens": 1930000, "total_tokens": 50140000, "total_cost_micros": 61240000, "last_completion_at": "2026-10-13T02:14:07Z" } ``` `total_cost_micros` is the stream's spend in millionths of a dollar — `61240000` is `$61.24`. Poll this instead of summing per-request costs yourself. ## Consume results with a cursor `GET /v1/responses/events` returns completions in order. Pass `stream_id` to read one stream's feed, or omit it for the account-wide feed of every background response. The cursor is `starting_after` (a completion id), so a consumer can resume exactly where it stopped: ```python from sference_sdk import SferenceClient client = SferenceClient() for event in client.iter_responses_events(stream_id=stream.id, consumer_name="worker-3"): process(event) ``` `iter_responses_events` persists the cursor per stream and consumer name via the SDK's checkpoint helpers, so a restarted worker picks up where it left off rather than replaying. Pass `from_latest=True` to skip history, or `checkpoint=False` to manage the cursor yourself with the lower-level `list_responses_events(stream_id=..., starting_after=...)`. This is what makes a stream safe to consume from a pool of workers: each consumer tracks its own position, so a duplicate is avoided by the cursor rather than by deduplicating after the fact. | Parameter | Meaning | | --- | --- | | `stream_id` | Read one stream's events. Omit for the account-wide feed. | | `limit` | Page size (default 20). | | `starting_after` | Cursor: return events after this completion id. | | `ending_before` | Cursor: return events before this completion id. | | `wait_ms` | Long-poll: hold the request open up to this many milliseconds for new events. | ## Lifecycle | Status | Meaning | | --- | --- | | `open` | Accepting new requests | | `cancelled` | Cancelled — pending items will not run | | `archived` | Closed for new work; results remain readable | ```bash # Rename curl -X PATCH https://api.sference.com/v1/streams/$STREAM_ID \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "nightly-eval-2026-10-13 (rerun)"}' # Cancel everything still pending curl -X PATCH https://api.sference.com/v1/streams/$STREAM_ID \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"status": "cancelled"}' # Archive when done curl -X PATCH https://api.sference.com/v1/streams/$STREAM_ID \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"status": "archived"}' ``` ## CLI ```bash sference stream create --name nightly-eval-2026-10-13 sference stream list sference stream status --stream-id $STREAM_ID sference stream submit --stream-id $STREAM_ID --model zai-org/GLM-5.3 --input prompt.txt sference stream archive --stream-id $STREAM_ID sference stream cancel --stream-id $STREAM_ID ``` ## Streams vs. batches Both group work on the 24h window, and they answer different questions. | | Stream | Batch | | --- | --- | --- | | Membership | by `metadata.stream_id` on any request | one uploaded JSONL file | | Submission | incremental, from anywhere | one create call | | Ordering | events endpoint with a resumable cursor | results file, one line per row | | Cancellation | cancel the stream | cancel the batch | | Best for | long-running pipelines, multi-worker consumers, jobs assembled over time | a fixed dataset submitted once | Use a batch when you have the whole request set up front. Use a stream when requests accumulate over time, when several consumers need to read results independently, or when you want a running cost rollup for a workload. - [Batch inference](https://sference.com/docs/guides/batches): The fixed-dataset alternative. - [Usage & analytics](https://sference.com/docs/usage): Attribute spend across streams, keys, and members. - [Python SDK](https://sference.com/docs/sdk): Checkpoint helpers for stream consumers. --- # Prompt caching > How sference reuses shared prompt prefixes, what you can do to hit the cache more often, and how to verify hits. Source: https://sference.com/docs/guides/prompt-caching Requests that share an opening are the norm in production: an agent that resends the same system prompt and tool definitions on every step, a chat that replays its history each turn, a document a user asks several questions about. Paying full price to reprocess that shared text every time is waste, and on long agent loops it is usually the largest line on the bill. sference matches the shared prefix for you and bills those tokens at the **cached input** rate, which is a fraction of the input rate on every model that offers one. Caching is **on by default**. There is nothing to enable. The rest of this page is about hitting it more often. ## Two rules decide whether a request hits 1. **A prefix matches byte for byte, from the very start of the prompt.** Matching stops at the first byte that differs. 2. **A cache is local to the worker that served the request.** A later request hits only if it lands on that same worker. You control the first rule completely. The second is what `prompt_cache_key` is for. ## Rule 1: keep your prefix stable Because matching starts at the beginning and stops at the first difference, the *order* of your prompt decides how much can be reused. Put what never changes at the front — system prompt, tool definitions, few-shot examples, a long document — and what varies at the end. Usually only the user's question changes, so it belongs last. > **Dynamic content at the start kills your hit rate** > > A timestamp, a request id, or a session id in the opening line changes the first bytes on every request, so nothing behind it can be reused — including a system prompt that never changed at all. This is the single most common cause of a 0% cache hit rate. ```python ❌ Don't — the timestamp changes the first bytes, so nothing after it is reused response = client.responses.create( model="zai-org/GLM-5.3", instructions=f"Current time: {datetime.now()}\n\n{SYSTEM_PROMPT}", input=question, ) ``` ```python ✅ Do — the prefix is identical every time; only the tail varies response = client.responses.create( model="zai-org/GLM-5.3", instructions=SYSTEM_PROMPT, input=f"{question}\n\nCurrent time: {datetime.now()}", ) ``` The same mistake wears other clothes: - A request id or trace id interpolated into the system prompt. - Tool definitions serialized in a different order between runs, or with object keys emitted in a different order. Serialize deterministically. - Retrieved documents or a conversation summary placed *before* the system prompt instead of after it. - A `datetime.now()` or `uuid4()` in a few-shot example. ## Rule 2: pin the worker with `prompt_cache_key` A prefix cache lives on one worker. If your next request lands on a different worker, it re-prefills from scratch even though the text is identical. `prompt_cache_key` solves this. Send a stable identifier for the sequence of requests that share a prefix — a chat session id, an agent run id, a document id — and sference routes every request with that key to the same worker while it stays warm. ```python response = client.responses.create( model="zai-org/GLM-5.3", instructions=SYSTEM_PROMPT, input=user_turn, prompt_cache_key=session_id, # same value for every turn of this session ) ``` ```bash curl https://api.sference.com/v1/chat/completions \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "messages": [{"role": "user", "content": "..."}], "prompt_cache_key": "sess_8f14e45f" }' ``` `prompt_cache_key` is accepted on `/v1/chat/completions` and `/v1/responses`. Without it, placement is inferred from the input text, which works well when prefixes are long and distinctive but is best-effort. > **Pick a key with the right lifetime** > > Use one key per conversation or agent run — not one per user, and not a global key. A key that is too broad spreads unrelated prefixes across one worker and wastes its cache; a key that changes every request defeats the purpose. If a session is idle long enough for the worker to be reassigned, the next request simply re-prefills once. ## What it costs Cached input bills at the **cached input** rate listed on the [Pricing](https://sference.com/docs/pricing) page — typically 10–25% of the input rate depending on the model. There is **no cache-write charge**: unlike some providers, we do not bill a premium for populating the cache, and no TTL choice changes the price. ## Verifying hits The token breakdown comes back on every response. On the OpenAI-shaped surfaces: ```json { "usage": { "prompt_tokens": 8420, "completion_tokens": 118, "prompt_tokens_details": { "cached_tokens": 8192 } } } ``` `prompt_tokens_details.cached_tokens` is the number of prompt tokens served from cache. If it is consistently `0` on requests you expect to hit, check, in order: 1. Is anything dynamic at the *start* of the prompt? (Rule 1 — the usual cause.) 2. Is `prompt_cache_key` set and stable across the requests? (Rule 2.) 3. Is the shared prefix long enough to be worth caching? Very short prefixes may not be cached at all. 4. Are consecutive requests arriving close together? A long idle gap can let the worker's cache evict. On `/v1/messages` the same numbers appear as `cache_read_input_tokens` in the Anthropic-shaped usage block. - [Pricing](https://sference.com/docs/pricing): Cached input rates per model, and how the windows stack. - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): Concurrency patterns that keep prefixes warm. - [Agent SDKs](https://sference.com/docs/agents): Agent loops are the workloads that benefit most from caching. --- # Images and vision > Send images alongside text to multimodal models on sference: which models accept images, the request shapes, limits, and how images are billed. Source: https://sference.com/docs/guides/vision Several catalog models accept images alongside text. Send a screenshot, a scanned document, a chart, or a product photo and ask about it — the same request shape works on `/v1/chat/completions`, `/v1/responses`, and `/v1/messages`. ## Which models accept images | Model | Vision | | --- | --- | | `moonshotai/Kimi-K3` | ✅ | | `zai-org/GLM-5.3-Flash` | ✅ | | `deepseek-ai/DeepSeek-V4.1-Flash` | ✅ | | `Cloudflare/clef` | ✅ | | `zai-org/GLM-5.3`, `zai-org/GLM-5.2` | ❌ text only | | `deepseek-ai/DeepSeek-V4-Flash`, `-0731` | ❌ text only | > **Capability comes from the catalog, not the model name** > > Send an image to a text-only model and the request fails — the platform does not silently drop it. Check `GET /v1/models`: each entry carries a `capabilities.image_input.supported` flag. Route image requests to a model where it is `true`. ## Sending an image ### Base64 data URL The portable form — works everywhere, no upload step: ```python import base64, os from openai import OpenAI client = OpenAI(base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"]) with open("invoice.png", "rb") as f: b64 = base64.b64encode(f.read()).decode() response = client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What is the total on this invoice?"}, {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}, ], }], ) ``` ### Remote URL If the image is already reachable over HTTPS, pass the URL directly: ```python {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}} ``` ### Responses API On `/v1/responses`, images are content parts of an `input` item: ```python response = client.responses.create( model="zai-org/GLM-5.3-Flash", input=[{ "role": "user", "content": [ {"type": "input_text", "text": "Summarize this chart."}, {"type": "input_image", "image_url": f"data:image/png;base64,{b64}"}, ], }], ) ``` ### Anthropic Messages API On `/v1/messages`, images are Anthropic content blocks: ```python { "role": "user", "content": [ {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}}, {"type": "text", "text": "What does this show?"}, ], } ``` ## Practical limits - **Formats:** PNG, JPEG, WebP, GIF. Convert exotic formats (HEIC, TIFF) before sending. - **Size:** keep images under ~5 MB. Large images are downscaled before tokenization, so sending a 20 MP photo costs bandwidth without improving accuracy. - **Resolution and tokens:** an image is converted to tokens, and the count scales with resolution. A full-page screenshot can cost more tokens than a page of text. Downscale to the smallest resolution that still answers your question — for a receipt, 1024 px on the long edge is usually plenty. - **Multiple images:** supported. Put the text instruction after the images it refers to, or the model may describe them without answering. ## Cost Images bill as **input tokens** at the model's normal input rate — there is no separate image surcharge. Because token count scales with resolution, image-heavy workloads are sensitive to the resolution you send. See [Pricing](https://sference.com/docs/pricing). ## Common patterns **Document extraction.** Screenshot or scan → vision model → [structured output](https://sference.com/docs/guides/structured-outputs) with a schema for the fields you need. This is the highest-value use of vision in production: invoices, receipts, forms, IDs. **UI and screenshot understanding.** Feed a screenshot to an agent so it can reason about layout without a browser tool. Note that coding models in the catalog are text-only — see [Claude Code](https://sference.com/docs/sference-code/claude-code) for how that is handled in an agent loop. **Chart and table reading.** Works well at moderate resolution; ask for the values explicitly rather than "describe this", and request JSON if you intend to compute on them. **Quality control.** Compare two images, or an image against a spec, with a yes/no or enum answer. - [Structured outputs](https://sference.com/docs/guides/structured-outputs): Pair vision with a schema for extraction workloads. - [Models](https://sference.com/docs/models): Check capabilities.image_input before routing image traffic. - [Pricing](https://sference.com/docs/pricing): Images bill as input tokens; resolution drives the count. --- # Max queue wait > Bound how long a request may wait behind queued work, and use the bound to fail over to another provider when sference is congested. Source: https://sference.com/docs/guides/max-queue-wait `max_queue_wait_s` is an **admission bound**. It states the maximum projected wait your request is willing to accept behind work that is already queued. If the scheduler cannot admit the request inside that budget, the request is rejected immediately with a `max_queue_wait_exceeded` error instead of sitting in a queue. That rejection is the point: it turns "this will be slow" from something you discover after a timeout into something you can act on in milliseconds. ```python response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": prompt}], max_queue_wait_s=15, ) ``` ## What the bound covers | Counts against the bound | Does not count | | --- | --- | | Projected wait behind already-queued work | Your own prompt's prefill time | | Pending time already spent waiting | Decode time after the first token | | For flex requests, the deliberate dispatch hold | Network transfer to and from the API | The bound is on **congestion**, not on prompt size. A 100k-token prompt on an idle fleet is admitted immediately; a short prompt behind a deep queue is not. Accepted range: **5–300 seconds**. Omit the field to use the platform default ceiling (60s for streaming requests, 300s for synchronous ones). ## Why you would set it ### Fast fallback to another provider This is the main use. Set a bound that reflects your real latency budget, and treat `max_queue_wait_exceeded` as a routing signal: retry the request against your fallback provider, or against a different sference model, rather than waiting. ```python from openai import OpenAI import openai sference = OpenAI(base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY) fallback = OpenAI(base_url="https://api.other-provider.com/v1", api_key=OTHER_KEY) def complete(prompt: str): try: return sference.chat.completions.create( model="zai-org/GLM-5.3", messages=[{"role": "user", "content": prompt}], max_queue_wait_s=15, ) except openai.BadRequestError as e: if "max_queue_wait_exceeded" in str(e): return fallback.chat.completions.create( model="gpt-5.6", messages=[{"role": "user", "content": prompt}], ) raise ``` ### Adopting sference as a preferred provider The bound is what makes sference safe to put **first** in a multi-provider chain. You get open-model economics on the overwhelming majority of requests, and a hard, fast signal to fall back when the fleet is busy — instead of a client timeout that wastes the whole budget before failing. Pair it with `service_tier: "flex"` when the request is not user-facing: flex requests count their deliberate dispatch hold against the bound, so a bound on a flex request bounds end-to-end scheduling latency rather than just queue depth. ### Protecting a batch of concurrent calls When you fan out hundreds of requests at once, a bound keeps a congested moment from turning into hundreds of connections held open for minutes. Requests that cannot be admitted fail fast and can be retried in a later wave. ## The error A rejected request returns a `400` with a structured body: ```json { "error": { "code": "max_queue_wait_exceeded", "message": "Projected wait exceeded max_queue_wait_s", "param": "max_queue_wait_s" } } ``` | Field | Meaning | | --- | --- | | `code` | `max_queue_wait_exceeded` | | `param` | `max_queue_wait_s` when **you** set the bound; `null` when the platform's default ceiling rejected it | | `message` | Human-readable detail | The distinction in `param` matters for monitoring: a rejection with `param: null` means the platform ceiling fired and you did not ask for a bound — lower your own bound to take control of that decision, or raise concurrency elsewhere. > **Rejected requests are not billed** > > A `max_queue_wait_exceeded` rejection happens before any inference work, so it costs nothing. Retrying it against a fallback is free on our side. ## Interaction with flex Flex requests are dispatched at lower priority by design, and that hold is counted against `max_queue_wait_s`. A flex request with a 30s bound will be rejected rather than held for the full flex window — which is usually what you want for a workload that has a deadline, and not what you want for a pure backfill job. For backfill, either omit the bound or use `background: true` with the 24h batch window, where the bound does not apply. ## Where it is accepted | Surface | Field | | --- | --- | | `/v1/chat/completions` | `max_queue_wait_s` | | `/v1/responses` | `max_queue_wait_s` | | `/v1/messages` | `max_queue_wait_s` | | Batch rows | Not supported — batch rows are already on the 24h window and cannot be rejected for congestion | - [Pricing](https://sference.com/docs/pricing): Flex and batch windows, and what they cost. - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): Concurrency patterns that use the bound well. - [Webhooks](https://sference.com/docs/guides/webhooks): For work you would rather not wait on at all. --- # Webhooks > Receive a callback when a background response finishes instead of polling for it. Source: https://sference.com/docs/guides/webhooks When a request can take a while to finish, you do not have to poll `GET /v1/responses/{response_id}` in a loop. Attach a callback URL to the request and sference delivers the finished result to your server as an HTTP POST. The feature works on `POST /v1/responses` and `POST /v1/chat/completions`, and is configured entirely through two `metadata` keys. > **Beta** > > Webhook delivery is in beta and enabled per account. If you set the metadata keys and no callback arrives, check with your account team — the request itself is never affected by webhook configuration. ## Parameters Both keys live inside the `metadata` object on the create request. | Key | Type | Meaning | | --- | --- | --- | | `metadata.completion_webhook` | string | Destination URL for the callback. Must be `http` or `https`. If omitted or unusable, no callback is sent and the request is unaffected. | | `metadata.webhook_token` | string | Optional shared secret. sference sends its value as `Authorization: Bearer ` so your endpoint can verify the call is genuine. | ## Set up an endpoint ### 1. Expose a route that accepts POST Your endpoint receives a JSON body and should respond as soon as it has accepted the payload. The body matches what `GET /v1/responses/{response_id}` returns for the same request, whether the response completed or failed. ```python from fastapi import FastAPI, Request app = FastAPI() @app.post("/hooks/inference-done") async def inference_done(request: Request): payload = await request.json() # payload is the same object GET /v1/responses/{id} returns enqueue_for_processing(payload["id"], payload) return {"ok": True} ``` ### 2. Verify the Authorization header If you set `webhook_token`, reject any call whose header does not match: ```python from fastapi import Header, HTTPException WEBHOOK_TOKEN = "whk_3f0a9c2e7b14" def assert_authorized(authorization: str = Header(default="")): if authorization != f"Bearer {WEBHOOK_TOKEN}": raise HTTPException(status_code=401) ``` ### 3. Submit a request that points at your endpoint Set `background: true` and add the callback URL (and token, if you use one) to `metadata`: ```python from openai import OpenAI client = OpenAI(base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY) response = client.responses.create( model="zai-org/GLM-5.3", input="Summarize this document.", background=True, metadata={ "completion_webhook": "https://app.example.com/hooks/inference-done", "webhook_token": "whk_3f0a9c2e7b14", }, ) ``` ## What sference sends A single POST request. | Part | Value | | --- | --- | | Method | `POST` to the URL in `completion_webhook` | | `Content-Type` | `application/json` | | `Authorization` | Present only when `webhook_token` was set: `Bearer ` | | Body | The full response object, identical to `GET /v1/responses/{response_id}` — including `status` and, for failures, the `error` envelope. The response `id` is in the body and is your deduplication key. | ## When a callback fires sference POSTs to your endpoint once a response reaches a terminal status: - **Completed** — the request finished normally. The body is the completed response object; for a Chat Completions request, the chat completion. - **Failed** — the body is the failed response object, including `error.code` and `error.message`. This includes a background request that did not finish before its deadline: it arrives with `status: "failed"` and `error.code: "timeout"`. Branch on `status` in your handler. ## Delivery semantics > **Callbacks can arrive more than once** > > Treat the response `id` in the body as an idempotency key and ignore any `id` you have already processed. - **Retries.** A delivery that returns a non-2xx status or hits a network error is retried up to **3 times**. Return a **2xx** as soon as you accept the payload to stop the retries. - **Timeout.** Each attempt has a **30 second** ceiling. A timeout counts as a failed attempt and triggers the next retry. - **Best effort.** If every attempt fails, the failure is logged but never affects the response record or the API. The result stays available through `GET /v1/responses/{response_id}` regardless of whether delivery ever succeeded. Because delivery is best-effort, a webhook is an optimization over polling, not a replacement for having the result retrievable. Keep the retrieval path in your code as the source of truth. ## Test it locally Point a real request at a listener on your own machine. ### 1. Run a listener that prints the body and returns 200 ```bash python -c " from http.server import HTTPServer, BaseHTTPRequestHandler; import json class H(BaseHTTPRequestHandler): def do_POST(self): print(json.dumps(json.loads(self.rfile.read(int(self.headers['Content-Length']))), indent=2)) self.send_response(200); self.end_headers() HTTPServer(('127.0.0.1', 8765), H).serve_forever() " ``` ### 2. Tunnel to it ```bash ngrok http 8765 ``` ### 3. Use the tunnel URL as `completion_webhook` Paste the `https://…ngrok-free.app` URL from ngrok into the `metadata` block above and submit the request. The listener prints the payload when the response finishes. - [Requests at scale](https://sference.com/docs/guides/requests-at-scale): When to fan out with webhooks vs. submit a batch. - [OpenAI Responses API](https://sference.com/docs/api/responses): Background responses, statuses, and the events API. - [Max queue wait](https://sference.com/docs/guides/max-queue-wait): Fail fast instead of waiting on a congested fleet. --- # Decisions > Realtime classification with POST /v1/decisions — ask up to 64 choice, score, or yes/no questions about one input and get calibrated probabilities back in a single call. Source: https://sference.com/docs/guides/decisions `POST /v1/decisions` answers structured questions about one input in a single realtime call. Instead of prompting a chat model and parsing its text, you describe the **questions** and their options. Back come **probabilities** for every option: routing, triage, moderation, intent detection, and tagging, with no output to parse. Decisions run on **[Clef](https://sference.com/docs/models#decisions-models)** (`Cloudflare/clef`), a 27B model built for this task. It reads the input and every question together and scores all options at once. It generates no text, so you pay for **input tokens only**. ## Request ```bash curl https://api.sference.com/v1/decisions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -d '{ "model": "Cloudflare/clef", "state": "I was charged twice for my subscription this month. Please refund the duplicate charge.", "questions": { "route": { "type": "choice", "criteria": { "billing": "Payments, invoices, refunds", "technical": "Bugs and outages", "sales": "Pricing and new plans" } }, "urgency": { "type": "score", "criteria": ["Low", "Medium", "High"] }, "refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" } } }' ``` - **`state`**: the input to decide about. It is either text or any JSON value, such as a ticket object or a conversation. The whole request is limited to 1 MiB, and the model has a 64K-token context. - **`questions`**: 1–64 named questions, all answered in the same call. Each may carry optional **`instructions`** that frame the question. | Type | `criteria` | Answer | | --- | --- | --- | | **`choice`** | Object of 1–256 labels → descriptions | `choice` (the most likely label), `confidence`, and `probabilities` per label | | **`score`** | Ordered list of 1–256 levels, lowest first | `score` (expected level index, 0-based), `confidence`, `legend`, and `probabilities` per level | | **`noul`** | Optional `{"true": …, "false": …}` descriptions | `noul`: the probability that the answer is yes | ## Response ```json { "model": "Cloudflare/clef", "answers": { "route": { "type": "choice", "choice": "billing", "confidence": 0.9909, "probabilities": { "billing": 0.9909, "technical": 0.0055, "sales": 0.0036 } }, "urgency": { "type": "score", "score": 1.5588, "confidence": 0.5892, "legend": { "0": "Low", "1": "Medium", "2": "High" }, "probabilities": { "0": 0.0304, "1": 0.3804, "2": 0.5892 } }, "refund": { "type": "noul", "noul": 0.9928 } }, "usage": { "input_tokens": 323, "output_tokens": 0 } } ``` The probabilities are the model's full distribution, not a single sampled answer. Pick your own thresholds per question: for example, auto-route above 0.9 and send anything lower to a human. ## Python SDK and CLI ```python from sference_sdk import ChoiceQuestion, NoulQuestion, ScoreQuestion, SferenceClient client = SferenceClient() # reads SFERENCE_API_KEY decision = client.create_decision( model="Cloudflare/clef", state="I was charged twice this month, please refund me.", questions={ "route": ChoiceQuestion(criteria={"billing": "Payments", "support": "Everything else"}), "urgency": ScoreQuestion(criteria=["Low", "Medium", "High"]), "refund": NoulQuestion(instructions="Does the customer ask for a refund?"), }, ) print(decision.answers["route"].choice, decision.answers["refund"].noul) ``` ```bash sference decisions create --model Cloudflare/clef \ --state "I was charged twice, please refund me" \ --questions '{"route": {"type": "choice", "criteria": {"billing": "Payments", "support": "Other"}}}' ``` `AsyncSferenceClient.create_decision` is the async equivalent. Questions may also be plain dicts with a `type`. With the CLI, `--state-json` sends a JSON state, and both options accept `@file` (or `@-` for stdin). ## Limits and errors Decisions are **realtime only**. They don't support batches, background responses, or the flex tier. | Status | Meaning | | --- | --- | | **400** | Invalid questions (count or criteria bounds), or a model that does not support decisions | | **402** | Insufficient credit or spending limit reached | | **413** | Request body over 1 MiB | | **429** | No decision capacity right now. Retry with backoff. | | **503** | Decisions temporarily unavailable. Honour `Retry-After`. | | **504** | Deadline passed while the request was running. **Not charged.** | Only successful decisions are billed, at $0.24 per 1M input tokens. The input count includes the rendered questions. --- # Requests at scale > Move tens of thousands of requests through sference without serializing them one by one. Source: https://sference.com/docs/guides/requests-at-scale A loop that fires requests sequentially stalls as soon as the workload grows past a few hundred calls. sference gives you two ways to clear large volumes: **fan out concurrent background responses**, or hand a single bundle to the **Batch API** and let sference work through it. ## Which path fits your workload - [Concurrent background responses](#fan-out-with-background-responses): You want results as each one finishes, need per-request control, or care about latency on individual items. You manage concurrency. - [Batch API](#submit-once-with-the-batch-api): You have a fixed set of requests and would rather submit once and collect later. sference owns the queue. | | Background responses | Batch API | | --- | --- | --- | | Unit of work | one HTTP request each | one JSONL file | | Results | poll each response id, or [webhook](https://sference.com/docs/guides/webhooks) | poll the batch, download JSONL | | Concurrency | you bound it | sference schedules it | | Per-item control | full — cancel, retry, inspect individually | limited to the batch | | Best for | dynamic workloads, agent fan-out, streaming results | fixed datasets, evals, offline jobs | | Window | 24h batch | 24h batch | The running example below is sentiment scoring for a backlog of product reviews. ## Fan out with background responses Submit each request with `background: true` and keep many in flight. For workloads of roughly 1,000 requests and up, four practices keep it stable: - **Use `AsyncOpenAI` with `DefaultAioHttpClient()`.** Under high concurrency the aiohttp backend outperforms the default `httpx` transport. - **Bound concurrency with an `asyncio.Semaphore`.** A fixed limit (200 is a sound starting point) caps simultaneous connections so you never exhaust them. - **Submit all requests first, then poll in a second pass.** Separating submission from polling keeps both phases simple and avoids interleaving. - **Set `max_queue_wait_s`** so a congested moment fails fast instead of holding connections open — see [Max queue wait](https://sference.com/docs/guides/max-queue-wait). ```bash pip install tqdm 'openai[aiohttp]' ``` ```python import asyncio from tqdm import tqdm from openai import AsyncOpenAI, DefaultAioHttpClient CONCURRENCY = 200 async def score(client, review: str) -> str: response = await client.responses.create( model="zai-org/GLM-5.3", input=f"Reply with one word, positive or negative:\n\n{review}", background=True, max_output_tokens=16, max_queue_wait_s=30, ) return response.id async def main(reviews: list[str]): async with AsyncOpenAI( base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY, http_client=DefaultAioHttpClient(), ) as client: sem = asyncio.Semaphore(CONCURRENCY) async def submit(review): async with sem: return await score(client, review) # Pass 1: submit everything, collect ids ids = [] for coro in tqdm(asyncio.as_completed([submit(r) for r in reviews]), total=len(reviews)): ids.append(await coro) # Pass 2: poll to completion results = {} pending = set(ids) while pending: for rid in list(pending): r = await client.responses.retrieve(rid) if r.status in ("completed", "failed", "incomplete"): results[rid] = r pending.discard(rid) if pending: await asyncio.sleep(1) return results asyncio.run(main(REVIEWS)) ``` > **Collect on the finish reason, not on status alone** > > On a reasoning model, `max_output_tokens` covers hidden reasoning as well as the answer. A response can settle `incomplete` with a perfectly good answer attached. Check the incomplete detail and use the output if it is there, rather than discarding it. ### Prefer webhooks to polling Polling every response id in a loop is the simplest thing that works, but it is wasteful at scale. If your workload is long-running and you have an endpoint to receive callbacks, register a [webhook](https://sference.com/docs/guides/webhooks) and let sference notify you when each response finishes. ## Submit once with the Batch API When the request set is fixed, the [Batch API](https://sference.com/docs/guides/batches) is simpler and cheaper than fanning out yourself: you upload one JSONL file, create a batch, and collect results when it completes. sference owns the queue and schedules the rows. ```python import json from openai import OpenAI client = OpenAI(base_url="https://api.sference.com/v1", api_key=SFERENCE_KEY) # 1. Write the JSONL — one request per line with open("reviews.jsonl", "w") as f: for i, review in enumerate(REVIEWS): f.write(json.dumps({ "custom_id": f"review-{i}", "method": "POST", "url": "/v1/chat/completions", "body": { "model": "zai-org/GLM-5.3", "messages": [{"role": "user", "content": f"Reply with one word, positive or negative:\n\n{review}"}], "max_tokens": 16, }, }) + "\n") # 2. Upload and create the batch uploaded = client.files.create(file=open("reviews.jsonl", "rb"), purpose="batch") batch = client.batches.create( input_file_id=uploaded.id, endpoint="/v1/chat/completions", completion_window="24h", ) # 3. Poll, then download while True: b = client.batches.retrieve(batch.id) if b.status in ("completed", "failed", "expired", "cancelled"): break time.sleep(10) content = client.files.content(b.output_file_id).text for line in content.splitlines(): row = json.loads(line) print(row["custom_id"], row["response"]["body"]["choices"][0]["message"]["content"]) ``` Full details — row body shapes, create-time validation, `custom_id` correlation, and the results endpoint — are in [Batch inference](https://sference.com/docs/guides/batches). ## Choosing between them - **Fixed dataset, no per-item interaction** → Batch API. Less code, sference schedules it, and you get one artifact at the end. - **Dynamic workload** — you are generating requests as results arrive, or a user is waiting on some of them → background responses with a concurrency bound. - **Need per-item cancellation or retry** → background responses. A batch is one unit. - **Tens of thousands of rows, results needed later** → Batch API. Uploading one file beats managing 50k connections. The two compose: use the Batch API for the bulk backfill and background responses for the interactive slice, on the same account and the same 24h window. - [Batch inference](https://sference.com/docs/guides/batches): JSONL jobs, body shapes, and create-time validation. - [Webhooks](https://sference.com/docs/guides/webhooks): Stop polling: get notified when work finishes. - [Max queue wait](https://sference.com/docs/guides/max-queue-wait): Fail fast under congestion instead of holding connections. --- # Structured outputs > Constrain a model's response to a JSON schema so you get parseable, predictable JSON on sference. Source: https://sference.com/docs/guides/structured-outputs Most production callers do not want prose — they want a JSON object they can parse. sference supports the OpenAI structured-output fields so you can constrain the response to a schema instead of hoping the model formats it correctly and repairing failures with regex. ## JSON schema Pass `response_format` with `type: "json_schema"` on `/v1/chat/completions`, or `text.format` on `/v1/responses`: ```python from openai import OpenAI client = OpenAI( base_url="https://api.sference.com/v1", api_key=os.environ["SFERENCE_API_KEY"], ) response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[ {"role": "system", "content": "Extract the fields from the support ticket."}, {"role": "user", "content": ticket_text}, ], response_format={ "type": "json_schema", "json_schema": { "name": "ticket", "strict": True, "schema": { "type": "object", "properties": { "severity": {"type": "string", "enum": ["low", "medium", "high"]}, "product": {"type": "string"}, "summary": {"type": "string"}, }, "required": ["severity", "product", "summary"], "additionalProperties": False, }, }, }, ) ticket = json.loads(response.choices[0].message.content) ``` On the Responses API the same schema goes in `text.format`: ```python response = client.responses.create( model="zai-org/GLM-5.3", input=ticket_text, text={ "format": { "type": "json_schema", "name": "ticket", "schema": {...}, "strict": True, } }, ) ``` ## JSON mode If you only need *valid JSON* and not a specific shape, `{"type": "json_object"}` is cheaper and less constrained. Remember to describe the shape in the prompt — JSON mode guarantees syntax, not structure: ```python response = client.chat.completions.create( model="zai-org/GLM-5.3", messages=[ {"role": "system", "content": "Reply with JSON: {\"sentiment\": \"positive\"|\"negative\", \"confidence\": 0-1}"}, {"role": "user", "content": review}, ], response_format={"type": "json_object"}, ) ``` ## Schema guidance Write schemas that are easy to satisfy, not just correct: - **Use `enum` for closed sets.** `{"type": "string", "enum": ["low", "medium", "high"]}` is far more reliable than asking for "a severity level". - **Set `additionalProperties: false` and list every field in `required`.** Optional fields invite the model to omit them inconsistently. - **Prefer flat objects.** Deeply nested schemas fail more often and are harder to debug. - **Avoid `oneOf`/`anyOf` where a plain optional field would do.** Unions are the most common source of malformed output. - **Keep field names and descriptions explicit.** The description is part of the prompt. > **Reasoning models and constrained output** > > On reasoning models, `max_output_tokens` covers hidden reasoning as well as the answer. If the cap is too low the response can settle `incomplete` with a partially-formed object. Collect on the finish reason rather than assuming a parseable body, and give reasoning models headroom above the size of the object you expect. ## Failure handling Even with a schema, treat the response as untrusted input: ```python try: data = json.loads(response.choices[0].message.content) except json.JSONDecodeError: # Retry once, or fall back to a repair pass ... ``` A constrained decode makes malformed JSON rare, not impossible — truncation from a token cap, an upstream error mid-stream, and model-specific quirks all still produce unparseable output occasionally. Validate against your schema before use, and retry on failure rather than writing the bad value. ## Where it is supported | Surface | Field | | --- | --- | | `/v1/chat/completions` | `response_format` (`json_schema`, `json_object`) | | `/v1/responses` | `text.format` | | `/v1/messages` | Not supported — Anthropic's Messages API has no equivalent field | | Batch rows | Same body shape as the surface the row targets | - [Agent SDKs](https://sference.com/docs/agents): Structured output as the return type of an agent tool. - [Vision](https://sference.com/docs/guides/vision): Schemas over images: extract fields from screenshots and documents. - [API reference](https://sference.com/docs/api-reference): Full field list for response_format and text.format. --- # SferenceCode > Route your coding agents through sference: one command per harness, open models behind the same agent loop, and per-engineer spend you can see. Source: https://sference.com/docs/sference-code **SferenceCode** runs your coding agents on open models. Claude Code, Pi, opencode, Codex, and Cursor keep working exactly as they do today — the harness, the tools, the workflow — but the model calls behind them go to sference instead of a closed provider. You get open-model token prices, a per-request audit trail, and per-engineer spend attribution. The agent loop is unchanged. ## Why route a coding agent Coding agents are the most token-hungry thing most teams run: a single refactor is hundreds of model calls, each resending a long system prompt and tool definitions. That is exactly the workload where open-model prices and prefix caching compound. | | Closed provider | SferenceCode | | --- | --- | --- | | Price per 1M tokens | frontier pricing | `0.20 / 0.60` (GLM-5.3-Flash) to `3.00 / 15.00` (Kimi-K3) | | Cached input | premium cache-write | no cache-write charge; cached input at 10–25% of input | | Per-engineer attribution | usually per key | per key, per harness, per engineer | | Model choice | the provider's lineup | any catalog model, or your own fine-tune | ## One command per harness ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh sference auth login --api-key 'sk_...' ``` Then launch your harness: | Harness | Command | Page | | --- | --- | --- | | **Claude Code** | `sference launch claude` | [Claude Code](https://sference.com/docs/sference-code/claude-code) | | **Pi** | `sference launch pi` | [Pi](https://sference.com/docs/sference-code/pi) | | **opencode** | `sference launch opencode` | [opencode](https://sference.com/docs/sference-code/opencode) | | **Codex CLI** | manual config | [Codex](https://sference.com/docs/sference-code/codex) | | **Cursor** | manual config | [Cursor](https://sference.com/docs/sference-code/cursor) | Each launcher writes the harness's own config file with a `sference` provider and starts it. Nothing global is changed, so your existing providers stay available. > **What the launchers write** > > `sference launch ` edits only that harness's config — `~/.pi/agent/models.json` for Pi, `~/.config/opencode/opencode.json` for opencode, and so on. Run it with `--dry-run` to print the config and command without launching anything. ## Choosing a model | Model | Input / output per 1M | Use for | | --- | --- | --- | | `zai-org/GLM-5.3-Flash` | 0.20 / 0.60 | Fast edits, formatting, small refactors, agent routing | | `zai-org/GLM-5.3` | 1.20 / 4.20 | General coding turns; the default for most harnesses | | `deepseek-ai/DeepSeek-V4.1-Flash` | 0.50 / 1.50 | Long-context work at low cost | | `moonshotai/Kimi-K3` | 3.00 / 15.00 | Hard planning turns, multi-file reasoning | Start on `GLM-5.3`, move routine turns to `Flash`, and reserve `Kimi-K3` for the turns that actually need it. Live ids and rates: `GET https://api.sference.com/v1/models`. ## What to expect **It is a compatibility layer.** Open models plan differently and call tools less reliably than frontier closed models on long repo work. It is genuinely useful for scripted runs, bulk refactors, and cost-sensitive loops. Validate on your own repository before committing a workflow. **Two calls per prompt is normal.** Claude Code fires a separate title-generation request alongside your turn. sference detects it, caps it, and forces reasoning off so it does not burn tokens. Seeing two requests per prompt is expected. **Thinking works, signatures do not.** Reasoning models stream `thinking` blocks that render normally, but `signature` is empty; multi-turn thinking continuity is best-effort. **Prefix caching is the main lever.** Every agent turn resends the system prompt and tool definitions. Keep them stable and first, and those tokens bill at the cached rate. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). ## Spend attribution Every request records the key that made it, so spend splits by engineer and by harness. `GET /control/v1/usage/by-key` and `GET /control/v1/activity` give the breakdown; see [Usage & analytics](https://sference.com/docs/usage). Give each engineer their own key (or their own harness profile) and the console's usage view answers "who spent what" without any instrumentation on your side. - [Claude Code](https://sference.com/docs/sference-code/claude-code): Proxy mode with hybrid routing: sference models alongside real Claude models. - [Pi](https://sference.com/docs/sference-code/pi): Provider block in ~/.pi/agent/models.json, picker scoped to sference. - [opencode](https://sference.com/docs/sference-code/opencode): sference provider in opencode.json with the full catalog in the picker. - [Usage & analytics](https://sference.com/docs/usage): Per-key and per-member spend attribution. --- # Claude Code > Run Claude Code against open models on sference: one-command launch with hybrid routing, manual ANTHROPIC_BASE_URL config, model recommendations, and known limits. Source: https://sference.com/docs/sference-code/claude-code [Claude Code](https://docs.anthropic.com/en/docs/claude-code) talks to the Anthropic Messages API, and sference [serves that API](https://sference.com/docs/api/messages) for open models. Point it at sference and your agent loop runs on `zai-org/GLM-5.3`, `moonshotai/Kimi-K3`, or your own fine-tune, at open-model token prices, with the same request-level audit trail as the rest of the platform. ## Fastest path: `sference launch claude` The [sference CLI](https://sference.com/docs/cli) ships a launcher that configures everything and starts Claude Code for you: ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh sference auth login sference launch claude ``` By default this runs in **proxy mode**: a local [mitmproxy](https://mitmproxy.org) forward proxy is started, Claude Code launches with `HTTPS_PROXY` pointed at it, and `ANTHROPIC_BASE_URL` is left unset so Claude Code's first-party detection stays on. The result is **hybrid routing**: sference models appear in Claude Code's `/model` picker and are routed to sference's `/v1/messages`, while real `claude-*` models pass straight through to Anthropic. You can switch between them mid-session. | Command | What it does | | --- | --- | | `sference launch claude` | Start the proxy and launch Claude Code. Catalog models are fetched live from `GET /v1/models`. | | `sference launch claude --model zai-org/GLM-5.3` | Put one specific model in the picker. | | `sference launch claude --models a,b` | Put a comma-separated set in the picker. | | `sference launch claude --dry-run` | Print the proxy config and command without launching. | | `sference launch claude --proxy-port 8082` | Pin the local proxy port (default: auto-pick). | | `sference launch claude -- -p "summarize this repo"` | Forward flags after `--` to Claude Code itself. | | `sference launch claude --no-anthropic` | Skip the proxy; route everything to sference via `ANTHROPIC_BASE_URL`. No hybrid routing, no picker. | Proxy mode needs `mitmdump` on your `PATH`. The installer adds it; on an existing install, `uv tool install mitmproxy`. If it is missing, the launcher tells you and suggests `--no-anthropic`. > **About the local CA** > > The proxy uses mitmproxy's local CA certificate, scoped to the Claude Code process via `NODE_EXTRA_CA_CERTS`. It is **not** added to your system trust store; only that one process trusts it, only for that session. Default model: `zai-org/GLM-5.3`. Override with `--model` or the `SFERENCE_MODEL` environment variable. ## Manual configuration If you would rather not run a proxy, configure Claude Code directly. Everything goes through `ANTHROPIC_BASE_URL`, so all traffic, including `claude-*` model names, hits sference, and the model is whatever you pin. Add to `~/.claude/settings.json` (or `.claude/settings.json` in a project): ```json { "env": { "ANTHROPIC_BASE_URL": "https://api.sference.com", "ANTHROPIC_AUTH_TOKEN": "sk_your_sference_key", "ANTHROPIC_MODEL": "zai-org/GLM-5.3" } } ``` - **Prefer `ANTHROPIC_AUTH_TOKEN`** (sent as `Authorization: Bearer`). `ANTHROPIC_API_KEY` also works (sference accepts `x-api-key`), but set one or the other, not both. - **`ANTHROPIC_MODEL` must be a sference catalog id.** There are no `claude-*` aliases; a Claude model name returns 400. See [Models](https://sference.com/docs/models). - **Restart Claude Code** after editing settings. Verify the endpoint independently of Claude Code: ```bash curl https://api.sference.com/v1/messages \ -H "Authorization: Bearer $SFERENCE_API_KEY" \ -H "content-type: application/json" \ -d '{ "model": "zai-org/GLM-5.3", "max_tokens": 256, "messages": [{ "role": "user", "content": "hi" }] }' ``` ## Choosing a model | Model | Why | | --- | --- | | `zai-org/GLM-5.3` | The CLI default. Strong general coding model, low latency; the current generation of the family. | | `moonshotai/Kimi-K3` | Largest reasoning model in the catalog; slower and pricier per token, for hard planning turns. | | Your fine-tune | [Custom models](https://sference.com/docs/custom-models) work on `/v1/messages` like any catalog model. | Live ids and pricing: `GET https://api.sference.com/v1/models`. ## What to expect **Two API calls per prompt is normal.** Claude Code fires a separate title-generation request alongside your actual turn: no tools, a `` user body, asking for a JSON title. sference detects it, caps it at 512 completion tokens, and forces reasoning off so it does not burn tokens on chain-of-thought before emitting a title. Seeing two requests per prompt in the console is expected, not a bug. **Thinking works, signatures do not.** Reasoning models stream `thinking` blocks that Claude Code renders normally, but `signature` is always empty. Multi-turn thinking continuity is best-effort: sference re-injects the inbound thinking text rather than a verified signed block. **No vision.** Catalog coding models are text-only. Screenshot tools and image attachments are replaced with a placeholder, and sference injects a system notice telling the agent it has no vision so it stops retrying image capture and verifies with `Read`, `grep`, and build output instead. If you specifically need image input, use `moonshotai/Kimi-K3`, `deepseek-ai/DeepSeek-V4.1-Flash` or `zai-org/GLM-5.3-Flash` — vision-capable models are general-purpose, not coding models. **Tool search may be off.** Claude Code disables tool search on non-Anthropic hosts unless told otherwise. In direct mode, `sference launch claude --no-anthropic --enable-tool-search` sets `ENABLE_TOOL_SEARCH=true` for you. ## Known limits | Limit | Effect | | --- | --- | | Empty thinking `signature` | Multi-turn thinking history degrades; nothing to verify or decrypt. | | No `redacted_thinking` | Anthropic's encrypted thinking blocks are never produced. | | `budget_tokens`, `thinking: adaptive`, `effort`, `display` | Accepted and ignored; bound output with `max_tokens`. | | No prompt-caching controls | `cache_control` blocks are ignored. Prefix caching is automatic; hits show up as `cache_read_input_tokens`. | | No server-side tools | MCP connectors, web search, code execution, and computer use are Anthropic-hosted and unavailable. Locally-run MCP servers configured in Claude Code work normally. | | Realtime only | No `service_tier: "flex"`, no background execution on `/v1/messages`. | | Model-dependent tool reliability | Open models call tools less consistently than Opus/Sonnet on long agent loops. | The full compatibility surface, covering every supported field, block type, and streaming event, is on the [Anthropic Messages API](https://sference.com/docs/api/messages) page. ## Troubleshooting **401 Unauthorized.** The key must be a sference `sk_...` key. If both `ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` are set, unset one. A stale Anthropic key in your shell profile overriding the sference key is the usual cause. **400 with an unknown-model message.** `ANTHROPIC_MODEL` is a Claude name, or a catalog id your key is not entitled to. Check `GET /v1/models`. **402 Payment Required.** The team credit balance is exhausted. Top up in the [console](https://app.sference.com). **Raw `` or `` text in replies instead of tool calls.** The model emitted tool syntax the parser did not extract, usually because `max_tokens` was too low and the call was truncated mid-JSON. Raise `max_tokens` (Claude Code tool turns need real headroom) or switch to a model with better tool support. **Slow first token.** Long time-to-first-token on a queued request is normal under load; the stream sends `ping` keepalives while waiting. A request that never completes within the 600s sync window returns 504 and is cancelled server-side. **504 Gateway Timeout.** A single turn exceeded the sync inference window. Shorten the turn, reduce context, or pick a faster model. ## Related - [Anthropic Messages API](https://sference.com/docs/api/messages): The endpoint itself: fields, streaming events, tool use, thinking, errors. - [CLI](https://sference.com/docs/cli): Install, auth, launch, batches, and streams. - [Models](https://sference.com/docs/models): Catalog ids, pinning, and picking a checkpoint. - [Custom models](https://sference.com/docs/custom-models): Run your own fine-tune behind the same endpoint. --- # Pi > Route Pi through sference: one command, a sference provider in ~/.pi/agent/models.json, and a model picker scoped to the sference catalog. Source: https://sference.com/docs/sference-code/pi [Pi](https://github.com/badlogic/pi-mono) reads providers from `~/.pi/agent/models.json`. The sference launcher writes a `sference` provider block there and starts Pi with it selected. ## Launch ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh sference auth login --api-key 'sk_...' sference launch pi ``` Pi starts with sference as the active provider and the full catalog in the model picker. ## What the launcher writes A `sference` provider in `~/.pi/agent/models.json`, using Pi's `openai-responses` API against sference's `/v1/responses`: ```json { "providers": { "sference": { "api": "openai-responses", "baseUrl": "https://api.sference.com/v1", "apiKey": "sk_...", "models": [ { "id": "zai-org/GLM-5.3", "name": "GLM 5.3", "reasoning": true, "input": ["text"] }, { "id": "moonshotai/Kimi-K3", "name": "Kimi K3", "reasoning": true, "input": ["text", "image"] } ] } } } ``` The model list is fetched live from `GET /v1/models`, so the picker mirrors your catalog exactly — including vision capability and context window per model. ## Options | Flag | Effect | | --- | --- | | `--model zai-org/GLM-5.3` | Select the active model (default: `zai-org/GLM-5.2` or `SFERENCE_MODEL`) | | `--base-url ` | Point at a different API base (default `https://api.sference.com`) | | `--dry-run` | Print the provider config and command without launching | | `sference launch pi -- /path/to/project` | Forward arguments after `--` to Pi | The launcher scopes Pi's picker to sference models by default (`--models sference/**`), so other providers you have configured stay hidden until you widen it. `**` is required because sference model ids contain a slash. ## Choosing a model | Model | Input / output per 1M | Use for | | --- | --- | --- | | `zai-org/GLM-5.3-Flash` | 0.20 / 0.60 | Fast edits, formatting, small refactors | | `zai-org/GLM-5.3` | 1.20 / 4.20 | General coding turns | | `deepseek-ai/DeepSeek-V4.1-Flash` | 0.50 / 1.50 | Long-context work at low cost | | `moonshotai/Kimi-K3` | 3.00 / 15.00 | Hard planning turns | ## Cost control - **Prefix caching.** Pi resends the system prompt and tool definitions every turn. Keep them stable and first so they bill at the cached rate — see [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **Cheap default, strong escalation.** Set `GLM-5.3-Flash` as the active model and switch to `Kimi-K3` for planning turns. - **Per-engineer attribution.** Each engineer's key shows up separately in `GET /control/v1/usage/by-key`. See [Usage & analytics](https://sference.com/docs/usage). > **It is a compatibility layer** > > Open models plan differently and call tools less reliably than frontier closed models on long repo work. Validate on your own repository before committing a workflow. See [SferenceCode](https://sference.com/docs/sference-code) for what to expect. - [SferenceCode](https://sference.com/docs/sference-code): All harnesses, model choices, and what to expect. - [Claude Code](https://sference.com/docs/sference-code/claude-code): Proxy mode with hybrid routing. - [opencode](https://sference.com/docs/sference-code/opencode): sference provider in opencode.json. --- # opencode > Route opencode through sference: a sference provider in opencode.json with the full catalog in the model picker. Source: https://sference.com/docs/sference-code/opencode [opencode](https://opencode.ai) reads providers from `~/.config/opencode/opencode.json`. The sference launcher writes a `sference` provider there and starts opencode with it selected. ## Launch ```bash curl -fsSL https://raw.githubusercontent.com/s-ference/sference/main/install.sh | sh sference auth login --api-key 'sk_...' sference launch opencode ``` ## What the launcher writes A `sference` provider using the AI SDK's OpenAI-compatible provider, with the full catalog registered: ```json { "provider": { "sference": { "npm": "@ai-sdk/openai-compatible", "name": "Sference", "options": { "baseURL": "https://api.sference.com/v1", "apiKey": "{env:SFERENCE_API_KEY}" }, "models": { "zai-org/GLM-5.3": { "name": "GLM 5.3" }, "moonshotai/Kimi-K3": { "name": "Kimi K3" } } } } } ``` The catalog is fetched live from `GET /v1/models`, so the picker matches your entitlements. The API key is injected at launch from `SFERENCE_API_KEY` rather than written into the file. ## Options | Flag | Effect | | --- | --- | | `--model zai-org/GLM-5.3` | Select the active model | | `--base-url ` | Point at a different API base | | `--dry-run` | Print the provider config and command without launching | | `sference launch opencode -- ` | Forward arguments to opencode | ## Reasoning levels opencode exposes reasoning effort as a model variant. sference models are addressed as `sference/`, and reasoning levels are selected through opencode's variant switcher rather than a separate model id — the launcher registers the base ids and opencode handles the level. ## Choosing a model | Model | Input / output per 1M | Use for | | --- | --- | --- | | `zai-org/GLM-5.3-Flash` | 0.20 / 0.60 | Fast edits, formatting, small refactors | | `zai-org/GLM-5.3` | 1.20 / 4.20 | General coding turns | | `deepseek-ai/DeepSeek-V4.1-Flash` | 0.50 / 1.50 | Long-context work at low cost | | `moonshotai/Kimi-K3` | 3.00 / 15.00 | Hard planning turns | ## Cost control - **Prefix caching** — keep the system prompt and tool definitions stable and first. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **Per-engineer attribution** — one key per engineer shows up separately in `GET /control/v1/usage/by-key`. See [Usage & analytics](https://sference.com/docs/usage). > **It is a compatibility layer** > > Open models plan differently and call tools less reliably than frontier closed models on long repo work. Validate on your own repository first. See [SferenceCode](https://sference.com/docs/sference-code) for what to expect. - [SferenceCode](https://sference.com/docs/sference-code): All harnesses, model choices, and what to expect. - [Pi](https://sference.com/docs/sference-code/pi): Provider block in ~/.pi/agent/models.json. - [Claude Code](https://sference.com/docs/sference-code/claude-code): Proxy mode with hybrid routing. --- # Codex CLI > Route the OpenAI Codex CLI through sference with a custom provider in ~/.codex/config.toml. Source: https://sference.com/docs/sference-code/codex The [Codex CLI](https://github.com/openai/codex) reads providers from `~/.codex/config.toml`. Add sference as a custom provider and select it as the active model provider. ## Configure Add to `~/.codex/config.toml`: ```toml model = "zai-org/GLM-5.3" model_provider = "sference" [model_providers.sference] name = "Sference" base_url = "https://api.sference.com/v1" env_key = "SFERENCE_API_KEY" wire_api = "chat" ``` Then export the key and run Codex: ```bash export SFERENCE_API_KEY="sk_..." codex ``` `env_key` names the environment variable Codex reads for the bearer token, so the key never lives in the config file. > **`wire_api`: chat or responses** > > `wire_api = "chat"` uses `/v1/chat/completions`; `wire_api = "responses"` uses `/v1/responses`. sference serves both. Use `responses` if your Codex version supports it — it is the surface Codex is built around and it unlocks background steps. ## Choosing a model ```toml model = "zai-org/GLM-5.3" # general coding turns # model = "zai-org/GLM-5.3-Flash" # fast edits, 0.20 / 0.60 per 1M # model = "moonshotai/Kimi-K3" # hard planning, 3.00 / 15.00 per 1M ``` Live ids and rates: `GET https://api.sference.com/v1/models`. ## Switching providers Keep your existing OpenAI configuration and switch by editing one line: ```toml model_provider = "sference" # or "openai" ``` Both providers stay configured, so you can move between them per session. ## Cost control - **Prefix caching.** Codex resends the system prompt and tool definitions every turn. Keep them stable and first so they bill at the cached rate. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **`max_output_tokens` on reasoning models** covers hidden reasoning, so cap it to avoid burning budget before the agent acts. - **Per-engineer attribution.** One key per engineer shows up separately in `GET /control/v1/usage/by-key`. See [Usage & analytics](https://sference.com/docs/usage). > **It is a compatibility layer** > > Open models plan differently and call tools less reliably than frontier closed models on long repo work. Validate on your own repository before committing a workflow. See [SferenceCode](https://sference.com/docs/sference-code) for what to expect. - [SferenceCode](https://sference.com/docs/sference-code): All harnesses, model choices, and what to expect. - [Claude Code](https://sference.com/docs/sference-code/claude-code): One-command launch with hybrid routing. - [API compatibility](https://sference.com/docs/compatibility): Which request fields sference honors. --- # Cursor > Route Cursor through sference using an OpenAI-compatible custom model, alongside Cursor's own models. Source: https://sference.com/docs/sference-code/cursor Cursor supports custom OpenAI-compatible models. Add sference as a provider and its models appear alongside Cursor's own, so you can pick per chat. ## Configure In Cursor: **Settings → Models → OpenAI API Key**, then enable **Override OpenAI Base URL**: | Field | Value | | --- | --- | | Base URL | `https://api.sference.com/v1` | | API Key | your `sk_...` sference key | | Model | a catalog id, e.g. `zai-org/GLM-5.3` | Add each model you want by name — Cursor does not enumerate the provider's catalog, so add the ids you use: ``` zai-org/GLM-5.3 zai-org/GLM-5.3-Flash deepseek-ai/DeepSeek-V4.1-Flash moonshotai/Kimi-K3 ``` > **Overriding the base URL affects all custom models** > > Cursor's override applies to the OpenAI-compatible provider as a whole. If you already use a custom OpenAI-compatible endpoint, switch between the two rather than expecting both to be active at once. Cursor's own hosted models are unaffected. ## Choosing a model | Model | Input / output per 1M | Use for | | --- | --- | --- | | `zai-org/GLM-5.3-Flash` | 0.20 / 0.60 | Inline completion, small edits, chat | | `zai-org/GLM-5.3` | 1.20 / 4.20 | General coding turns, agent mode | | `deepseek-ai/DeepSeek-V4.1-Flash` | 0.50 / 1.50 | Long-context work at low cost | | `moonshotai/Kimi-K3` | 3.00 / 15.00 | Hard planning turns | Live ids and rates: `GET https://api.sference.com/v1/models`. ## Cost control - **Prefix caching** is the main lever — Cursor resends the system prompt and file context every turn. Keep the stable part first. See [Prompt caching](https://sference.com/docs/guides/prompt-caching). - **Use Flash for inline work.** Tab completion and small edits do not need a frontier model. - **Per-engineer attribution.** One key per engineer shows up separately in `GET /control/v1/usage/by-key`. See [Usage & analytics](https://sference.com/docs/usage). > **Vision** > > Some catalog models accept images. Cursor's image attachments go through as content parts, so a vision-capable model can read screenshots. Check `capabilities.image_input.supported` on `GET /v1/models` before relying on it. See [Images and vision](https://sference.com/docs/guides/vision). - [SferenceCode](https://sference.com/docs/sference-code): All harnesses, model choices, and what to expect. - [Codex CLI](https://sference.com/docs/sference-code/codex): Custom provider in ~/.codex/config.toml. - [Images and vision](https://sference.com/docs/guides/vision): Which models accept images, and how they bill. ---