It's a Tuesday afternoon and your primary model provider starts returning 503 overloaded. Not for everyone, not for long, but for twenty minutes your support agent, your invoice agent and the workflow that screens job applications all hit the same wall.
There are three cheap answers, and each one goes wrong in its own way. Retry the same provider harder and you add backoff on top of an outage, so every user waits and then fails anyway. Fall back on any error and a request that was simply malformed gets answered by a weaker model, which hides your bug behind a plausible reply. Fall back to a smaller model without checking the fit and the provider rejects the request, or something trims the conversation to make it fit, and the agent answers without the question the user just asked.
This post is a hands-on tour of how Algen Agent Runtime routes a model call across providers. We'll configure a three-provider chain, then break the primary on purpose, eleven different ways, and watch what the router does each time. Everything runs without an API key.
The short version
- Declare a chain, not a provider. An agent has a default model and an ordered list of fallbacks, each pointing at a named provider instance.
- Fall back only on failures specific to one provider. Outages, timeouts, rate limits, auth failures and a model the provider doesn't serve move on. A request the provider rejects as invalid doesn't, because every provider would reject it.
- Stop calling a provider that keeps failing. Each provider has a circuit breaker. Once it opens, the router skips that provider entirely until a reset period has passed.
- Pack the request separately for each model. A fallback with a smaller context window gets its own packing of the original request. If what must be sent doesn't fit, that candidate is skipped without a call.
- Record which model answered. Every run's
model.completedevent names the provider and model, so a fallback is never invisible.
Set up a three-provider chain
The runtime ships an example, examples/pattern_provider_fallback, with Anthropic as the primary, OpenAI as a cloud fallback and a local Ollama model as the last resort. Its agent.yaml is short:
providers:
primary:
type: anthropic
api_key: env://ANTHROPIC_API_KEY
default_model: claude-sonnet-5-5
fallback:
type: openai
api_key: env://OPENAI_API_KEY
default_model: gpt-5
local:
type: ollama
base_url: http://127.0.0.1:11434/v1
default_model: qwen3:8b
agents:
- name: resilient
version: 1.0.0
system_instructions: Produce a concise, verified response.
default_model:
name: quality
provider: primary
model: claude-sonnet-5-5
quality_tier: 3
context_window_tokens: 32000
context_safety_margin_tokens: 512
fallback_models:
- name: cloud-fallback
provider: fallback
model: gpt-5
quality_tier: 2
context_window_tokens: 16000
context_safety_margin_tokens: 512
- name: local-fallback
provider: local
model: "qwen3:8b"
quality_tier: 1
context_window_tokens: 12000
context_safety_margin_tokens: 512
budget:
max_output_tokens: 4096
A few things are worth noticing before we run anything.
Keys under providers: are instance names, not vendors. primary, fallback and local are the names the agent refers to; type: picks the adapter. You can run two Ollama instances, or two Azure deployments, and each gets its own health tracking, rate limit, circuit breaker and cost accounting.
Secrets are references. env://ANTHROPIC_API_KEY is resolved just before a call, and a literal key in the file is rejected when the settings load.
Typos fail at load time. If a fallback names a provider that isn't configured, load_settings() raises a ConfigurationError naming the exact path. A misspelt fallback never sits unnoticed until the day you need it.
Context windows are declared, not guessed. The runtime doesn't infer a window from a model name. The context_window_tokens values here are deployment choices; set them to what your deployment actually supports.
Run it with real keys and you'll see which route answered:
pip install -e '.[dev]'
export ANTHROPIC_API_KEY=... # primary
export OPENAI_API_KEY=... # cloud fallback
python -m examples.pattern_provider_fallback.app "What year did the first moon landing occur?"
[Route: anthropic/claude-sonnet-5-5]
The first moon landing occurred in 1969, when Apollo 11 landed on the lunar surface on July 20th.
Or, with no keys at all, run the same router with a primary that fails on purpose:
python -m examples.pattern_provider_fallback.app --offline
[offline: the primary provider fails on purpose; the router falls back to mock]
[Route: mock/deterministic]
deterministic fallback response
One call, three candidates
Here is what happens inside one model call when the primary is down.
The router builds a list of candidates from the default model and the fallbacks, drops any that can't serve the request, then tries them in turn until one answers. Two things decide whether a failure moves on to the next candidate or ends the call.
Which failures fall back
The question the router asks of every error is: would another provider fail the same way?
| The provider says | Falls back? | Why |
|---|---|---|
| 5xx, transport error, overloaded | Yes | This provider is having a bad time |
| Timeout, including HTTP 408 | Yes | Same |
| 429 rate limit | Yes | Your quota there is used up, not everywhere |
| 401 / 403 | Yes | A rotated or revoked key is specific to that provider |
| 404 for a model it doesn't serve | Yes | Another provider may serve its own model |
| Invalid or unparseable response | Yes | That model misbehaved |
| 400 / 422 invalid request | No | Every provider would reject the same request |
The last row is the important one. A 400 usually means your request is broken: a tool schema the API rejects, an unsupported parameter, a malformed message. Falling back would get you an answer from the next model, possibly a worse one, and the bug would ship. So the router raises the error as it is, and doesn't count it against the provider's health either, because the provider did nothing wrong.
One case looks like a provider failure but isn't: a missing API key. The key is resolved before the call, so a missing ANTHROPIC_API_KEY is a configuration error and stops the run rather than quietly routing everything to the fallback for weeks.
Break it on purpose
The router is plain Python, so you can drive it directly with a fake provider and watch every decision. This script uses the runtime's real ModelRouter and its mock provider. Save it anywhere and run it from a checkout of the runtime:
import asyncio
import time
from algen_agent_runtime.exceptions.errors import BudgetExceededError, ProviderError
from algen_agent_runtime.models.base import ModelRouter
from algen_agent_runtime.models.providers.mock import MockModelProvider
from algen_agent_runtime.types.contracts import (
ErrorKind, Message, ModelCapabilities, ModelProfile, ModelRequest, Role,
)
class FlakyProvider:
"""A provider that fails however we tell it to; failure=None means healthy."""
def __init__(self, provider_id: str) -> None:
self.provider_id = provider_id
self.failure: ProviderError | None = None
self.calls = 0
async def capabilities(self, model: str) -> ModelCapabilities:
return ModelCapabilities(chat=True)
async def generate(self, request: ModelRequest):
self.calls += 1
if self.failure:
raise self.failure
response = await MockModelProvider(responses=("ok",)).generate(request)
return response.model_copy(update={"provider": self.provider_id})
async def health(self) -> bool:
return self.failure is None
Three failures to throw, in the shape a real adapter raises them:
UNAVAILABLE = ProviderError("503 overloaded", ErrorKind.UNAVAILABLE, True, status_code=503)
BAD_REQUEST = ProviderError("400 bad tool schema", ErrorKind.INVALID_REQUEST, status_code=400)
NOT_SERVED = ProviderError("404 no such model", ErrorKind.INVALID_REQUEST, status_code=404)
A primary with a 32k window and a fallback with 16k, tried in declaration order. The circuit breaker is tightened to three failures and a two-second reset so the demo runs quickly; the defaults are five failures and 30 seconds.
PROFILES = (
ModelProfile(name="quality", provider="primary", model="big",
routing_preference="ordered", context_window_tokens=32_000),
ModelProfile(name="cloud-fallback", provider="fallback", model="mid",
routing_preference="ordered", context_window_tokens=16_000),
)
primary, fallback = FlakyProvider("primary"), FlakyProvider("fallback")
router = ModelRouter(circuit_failure_threshold=3, circuit_reset_seconds=2)
router.register_provider(primary)
router.register_provider(fallback)
async def call(label: str, failure: ProviderError | None, text: str = "hi") -> None:
primary.failure = failure
before = primary.calls
request = ModelRequest(messages=(Message.text(Role.USER, text),), max_output_tokens=1024)
try:
response = await router.generate(request, PROFILES)
result = f"-> {response.provider}/{response.model}"
except (ProviderError, BudgetExceededError) as exc:
result = "raised: " + str(exc).split("; ")[-1]
tried = "tried" if primary.calls > before else "skipped"
print(f"{label:<16} primary {tried:<8} {result}")
async def main() -> None:
for n in range(1, 6):
await call(f"{n}. 503", UNAVAILABLE) # the circuit opens after the third
await call("6. recovered", None) # still open: not tried
time.sleep(2.1)
await call("7. after reset", UNAVAILABLE) # tried once, fails, reopens at once
time.sleep(2.1)
await call("8. after reset", None) # tried, succeeds, circuit closes
await call("9. 400 request", BAD_REQUEST)
await call("10. 404 model", NOT_SERVED)
await call("11. 20k prompt", UNAVAILABLE, "log line " * 9_000)
asyncio.run(main())
The output, from runtime 0.1.0a23:
1. 503 primary tried -> fallback/mid
2. 503 primary tried -> fallback/mid
3. 503 primary tried -> fallback/mid
4. 503 primary skipped -> fallback/mid
5. 503 primary skipped -> fallback/mid
6. recovered primary skipped -> fallback/mid
7. after reset primary tried -> fallback/mid
8. after reset primary tried -> primary/big
9. 400 request primary tried raised: 400 bad tool schema
10. 404 model primary tried -> fallback/mid
11. 20k prompt primary tried raised: fallback/mid: configured context budget exceeded
Every line is a decision worth understanding. Let's go through them in groups.
Calls 1 to 8: the circuit breaker
The first three calls try the primary, get a 503 and fall back. That's correct but slow: during an outage, every request pays for a doomed call to the primary before it reaches a model that works. If the primary's failure is a timeout, that's the whole timeout, every time.
So after a run of consecutive failures, the primary's circuit opens, and calls 4 to 6 don't call it at all. Call 6 is the interesting one: the primary has actually recovered, but the router doesn't know that and doesn't check. It keeps sending traffic to the fallback until the reset period has passed.
After the reset (call 7), the primary is a candidate again, but on probation. Its failure count wasn't cleared, so a single further failure reopens the circuit straight away rather than allowing another three. Only a success (call 8) closes the circuit and resets the count.
A few properties fall out of how this is built:
- Health is per provider instance. Two Azure deployments of the same model have separate breakers, so one bad region doesn't take out the other.
- A malformed tool call isn't an outage. If the model answers but writes a tool call that doesn't parse, the router moves on without charging the provider's health; the provider worked, its model just made a mistake.
- A rejected request isn't an outage either. Calls that fail with a 400 or 422 never count towards opening the circuit.
Call 9: the request that fails everywhere
The primary rejects the request with a 400, and the router raises it. The fallback is never called. Your logs show the real error, 400 bad tool schema, instead of a successful answer from a different model with your bug still in place.
Call 10: the 404 that doesn't
A 404 looks like a client error, but here it means "I don't serve that model", which says nothing about whether the fallback serves its model. So it falls back. This catches a real class of incident: a provider retires a model version, or a deployment name in one region is wrong.
The distinction is made on both the error kind and the status code. An invalid_request with status 400 or 422 (or no status at all) stops; anything else that's provider-specific moves on.
Call 11: when the fallback is too small
The last call sends a prompt of roughly 20k tokens while the primary is down. The fallback has a 16k window, so the router has a choice: send it anyway and let the provider reject it, cut the conversation until it fits, or refuse.
It refuses, and it does so for a precise reason. Before every dispatch, the router computes how much room there is for messages on that candidate:
message allowance = context window
- requested output tokens
- safety margin
- estimated tool definitions, response schema and extensions
It then packs the request into that allowance. The system message, the latest user message and the newest group of messages are protected. Tool calls stay together with their results. Older groups can be dropped, oldest first. The output reservation is never silently shrunk to make room.
Each candidate is packed from the original request, not from what the previous candidate received. A small primary can't strip history that a larger fallback would have kept.
If the protected part doesn't fit, the candidate is skipped before anything is spent: no rate-limit slot, no metered credits, no call and no mark against its health. In call 11 the primary failed and the fallback couldn't fit, so the router raised an error naming both reasons. That's the right outcome. An agent answering a 20k-token question with the question cut off would be worse than an error.
When the primary is healthy, the same 20k prompt goes to the 32k primary as normal. Budgets decide where a request can go; they don't change where it goes first.
The estimate is based on the size of the serialised content, not a provider's tokenizer, which is why the safety margin exists. If you see "context budget exceeded" for prompts that should fit, the declared window is usually smaller than the deployment really supports.
Ordered or balanced
The demo used routing_preference: ordered, which tries candidates strictly in the order you declared them. Providers with an open circuit, missing capabilities or exceeded latency or cost limits are still skipped.
default_model:
name: primary
provider: gemini
model: gemini-flash-latest
routing_preference: ordered
fallback_models:
- name: fast
provider: groq
model: openai/gpt-oss-120b
routing_preference: ordered
ordered describes the whole list, so configuration that sets it on only some of an agent's models is rejected when the settings load.
The default, balanced, ranks candidates by a score instead: 1,000 points per quality_tier, minus the provider's last observed latency in milliseconds, minus its average cost per 1k tokens. One tier is worth a full second of latency. That means a primary that's up but slow can lose its place: at tier 3 and 3.5 seconds it scores −500, while a tier 2 fallback answering in 1.2 seconds scores 800 and is tried first.
That can be exactly what you want for an interactive agent, and exactly what you don't want when the fallback is a cheaper, weaker model. Pick deliberately:
orderedwhen the fallbacks are there for outages only and quality must not drift.balancedwhen the candidates are roughly interchangeable and latency matters most.local_first/cloud_firstwhen locality matters more than anything else, for example keeping data on a local model whenever one is healthy.
You can also make a candidate ineligible outright: required_capabilities (such as tools or json_schema) removes models that can't do the job, and max_latency_ms or max_cost_per_1k_tokens removes ones that are too slow or too expensive.
Retries sit outside fallback
Fallback and retries are separate layers, and it helps to know which does what.
The router works through the candidate list once per attempt. If every candidate fails and all of those failures are retryable, the runtime retries the whole chain with exponential backoff, according to the agent's retry_policy:
agents:
- name: resilient
retry_policy:
max_attempts: 3 # the default
initial_backoff_seconds: 0.1
max_backoff_seconds: 5
Each retry is recorded as a model.retrying event with the attempt number, the error kind and the delay. A failure that isn't retryable, such as the 400 in call 9, ends the attempt without a retry.
Streaming is stricter. Before a streamed call starts, the router can skip a candidate whose window is too small for the request. After that it's committed: a provider error doesn't fall back, and it never switches providers once tokens have started arriving. Replaying half an answer from a different model would be worse than failing.
Know which model answered
Fallback that you can't see is a quality problem waiting to be found. Every model call in a run emits model.completed with the provider and model that actually answered, along with tokens, latency and estimated cost. The example reads it back like this:
completed = [
event.data
for event in await container.runtime.events.history(result.run_id)
if event.type == "model.completed"
]
route = f"{completed[-1]['provider']}/{completed[-1]['model']}" if completed else None
Send these events to your tracing (Traccia, or any OpenTelemetry backend) and you can answer "how much of yesterday's traffic ran on the fallback?" with a query rather than a guess. Because metering happens per call, the credit metering for a fallback call is charged against the model that actually ran, at its own price.
Bring your own provider
Any provider can join a chain, as long as it reports failures in the router's terms. Implement the ModelProvider protocol and translate the vendor's errors into a ProviderError with an ErrorKind, an honest retryable flag and the HTTP status. A sketch for an HTTP-based adapter:
except httpx.HTTPStatusError as exc:
status = exc.response.status_code
kind = {
401: ErrorKind.AUTHENTICATION,
403: ErrorKind.AUTHORIZATION,
408: ErrorKind.TIMEOUT,
429: ErrorKind.RATE_LIMIT,
}.get(status, ErrorKind.UNAVAILABLE if status >= 500 else ErrorKind.INVALID_REQUEST)
raise ProviderError(
f"{self.provider_id} returned {status}",
kind,
retryable=status in {408, 429} or status >= 500,
status_code=status,
) from exc
Getting this mapping right is most of the work. Report a provider outage as INVALID_REQUEST with status 400 and the router will never fall back from it; report a broken request as UNAVAILABLE and every bug becomes a silent fallback. OpenAI-compatible endpoints can reuse OpenAICompatibleProvider, which already does the translation, and register with type: openai_compatible and a base_url.
What this doesn't solve
The circuit breaker is per process. Each runtime process keeps its own provider health in memory. Ten replicas each learn about an outage separately, and each pays its own failed calls before its circuit opens.
The breaker settings aren't in agent.yaml yet. The threshold (5) and reset period (30 seconds) are fixed when the service builds its router. Code that constructs ModelRouter directly can change them, as the demo does.
There's no single trial request after a reset. When the reset period ends, every call in flight may try the primary at once, not just one. If it's still down, they all fail over, and the circuit reopens.
A fallback is a different model. The runtime makes sure the fallback can do the job (capabilities, context window, cost and latency limits) and records that it ran. It can't make it answer as well as the primary. Evaluate your agent on each model in its chain, not just the first.
The token estimate is approximate, and configured budgets currently support text messages only; an image or file block with a budget set is a configuration error.
It's early. At the time of writing the runtime is 0.1.0a23, a pre-release.
Five things worth copying
- Fall back on the provider's failure, never on the request's. A 400 that falls back turns a bug into a quality regression nobody notices.
- Stop calling what's down. A circuit breaker turns an outage from "every request is slow" into "every request goes straight to the fallback".
- Pack for each model separately, protect the user's latest message, and skip a model rather than cut the question to fit.
- Make the fallback order a decision. Strict order for outage-only fallbacks; a score only when the models are truly interchangeable.
- Record which model answered, every time, next to what it cost.
An outage at your model provider is going to happen. With a declared chain, a breaker and an honest record of who answered, it becomes a dip on a dashboard rather than an incident.