Model fallback
Point sference at any OpenAI-compatible endpoint and key, and requests we can't serve (503 / capacity) fail over to it automatically — no code changes.
Model fallback
Model fallback keeps a request working when sference can't serve it: the model's queue is full, its admission ceiling fired, or the worker holding it died before the first token. Instead of a 503, the request is retried server-side against your own fallback provider — and you get one response, from whichever side served it.
You configure it once, and it applies to every request on every API key of your organization, on /v1/chat/completions, /v1/responses, and /v1/messages. No code changes.
The fallback endpoint (console setup)
Console → Members → the Organization card → Fallback endpoint:
| Field | Notes |
|---|---|
| Endpoint URL | Any OpenAI-compatible base URL — OpenRouter, Together, Fireworks, or your own vLLM. https:// only. |
| API key | Your own key at that provider. Shown once, then masked; stored encrypted. |
The requested model id is forwarded to your endpoint as-is — whether it accepts that name is between you and your provider (OpenRouter mostly understands our catalog ids; your own vLLM can be configured to).
Test connection fires a 1-token request so a typo is caught at save time, not at 3am. Removing the endpoint disables fallback.
Your prompts leave our fleet
A fallback sends the full request — messages, images, everything — to the endpoint you configured, under your agreement with that provider. It is not covered by our EU residency or our data-processing terms.
ZDR organizations cannot set a fallback endpoint. If your org has zero data retention enabled, the option is unavailable and external fallback is refused.
When fallback fires
Fallback happens only before any response byte has been sent, and only on capacity — when we cannot serve the model at all:
| Trigger | What happened on our side |
|---|---|
| Queue full | The model's pending queue is at its platform cap (model_at_capacity) |
| Admission ceiling | The scheduler projected the wait would exceed the model's TTFT ceiling (max_queue_wait_exceeded) |
| Worker lost | The worker holding the request died or stopped responding before the first token |
Fallback never fires on:
- Your own
max_queue_wait_sbound firing (429). You set an explicit latency budget; silently trying another provider would spend it somewhere you didn't ask. - Request errors (4xx). Schema lint, policy, balance, an org allowlist rejection, or
context_length_exceeded— those are properties of the request, not capacity, and every provider would fail them the same way. - Anything after the first token. No mid-stream failovers, ever.
- Batches, embeddings, background responses, decisions. The 24h window has no admission ceiling, so there is nothing to fall back from.
If the fallback endpoint also fails, you get a 503 with the fallback headers set, so you can tell both sides failed rather than just sference.
What you get back
The response model is always the model that served — for the fallback endpoint, the id it returned. Never hidden.
When a request fell back, two headers tell you so:
| Header | Example | Meaning |
|---|---|---|
x-sference-served-model | moonshotai/Kimi-K3 | The model id the endpoint served |
x-sference-fallback-index | 1 | 0 = sference served it; 1 = your fallback endpoint did |
Billing: a request served by your fallback endpoint bills $0 from sference — it's your key at your provider account, and your provider bills their share on it. Token counts are recorded so your usage page stays complete, and the failed first attempt costs nothing.
Seeing which requests fell back
Every fallback is visible in your own data. On the console requests page, a request that fell back carries a fallback badge — requested model → served model, and what triggered it (queue full, admission ceiling, worker lost) — and its cost shows $0. The same fields are on the requests API, so you can list every fallback over a period and reconcile them against your provider's bill.
Fallback and prompt caching
A fallback lands on a different provider, which means a cold prompt cache — the first served request there runs a full prefill at their rates. Fallback trades cache warmth for availability. If cache hits matter more than failover, prefer prompt_cache_key with a single model; see Prompt caching.
Interaction with other features
| Feature | Behavior |
|---|---|
service_tier: "flex" | Flex requests have no admission ceiling and bypass the queue cap, so they never trigger fallback. |
service_tier: "priority" | Tried at priority first; fallback still fires on worker loss. |
max_queue_wait_s | A client-bound rejection (429) ends the request — no fallback. See above. |
Idempotency-Key | The key resolves to the request that served, whichever side that was. A replay returns the original outcome. |
| Images | Forwarded as externalized URLs — your endpoint must be able to fetch them over the internet. |
Choosing a setup
- "Never fail my users" — point the endpoint at a large aggregator (OpenRouter) or a second provider with a model close to your primary. You're covered from the next request on.
- "I want my own GPUs as the net" — point it at your own vLLM/SGLang deployment serving the same weights. Same model id, same behavior, your hardware.
- "The exact model matters" — skip fallback and use the manual pattern in Max queue wait: fail fast, then decide yourself — another provider, a queue, or a human.
Fallback changes who serves the answer — and with another provider, potentially the answer's style, tool-calling reliability, and reasoning behavior. It's for workloads where "some good answer now" beats "the exact model later".