Skip to content
    Back to insights
    AI EngineeringResearch note

    When the Model Provider Goes Down: Fallback Routing for AI Agents

    Your primary model provider starts returning 503s. Retry harder and every user waits; fall back on any error and your bugs hide behind a weaker model. A hands-on tour of how the runtime routes around an outage, with a script that breaks the primary eleven ways.

    Saurav Samantray12 min read

    It's a Tuesday afternoon and your primary model provider starts returning 503 overloaded. Not for everyone, not for long, but for twenty minutes your support agent, your invoice agent and the workflow that screens job applications all hit the same wall.

    There are three cheap answers, and each one goes wrong in its own way. Retry the same provider harder and you add backoff on top of an outage, so every user waits and then fails anyway. Fall back on any error and a request that was simply malformed gets answered by a weaker model, which hides your bug behind a plausible reply. Fall back to a smaller model without checking the fit and the provider rejects the request, or something trims the conversation to make it fit, and the agent answers without the question the user just asked.

    This post is a hands-on tour of how Algen Agent Runtime routes a model call across providers. We'll configure a three-provider chain, then break the primary on purpose, eleven different ways, and watch what the router does each time. Everything runs without an API key.

    The short version

    1. Declare a chain, not a provider. An agent has a default model and an ordered list of fallbacks, each pointing at a named provider instance.
    2. Fall back only on failures specific to one provider. Outages, timeouts, rate limits, auth failures and a model the provider doesn't serve move on. A request the provider rejects as invalid doesn't, because every provider would reject it.
    3. Stop calling a provider that keeps failing. Each provider has a circuit breaker. Once it opens, the router skips that provider entirely until a reset period has passed.
    4. Pack the request separately for each model. A fallback with a smaller context window gets its own packing of the original request. If what must be sent doesn't fit, that candidate is skipped without a call.
    5. Record which model answered. Every run's model.completed event names the provider and model, so a fallback is never invisible.

    Set up a three-provider chain

    The runtime ships an example, examples/pattern_provider_fallback, with Anthropic as the primary, OpenAI as a cloud fallback and a local Ollama model as the last resort. Its agent.yaml is short:

    providers:
      primary:
        type: anthropic
        api_key: env://ANTHROPIC_API_KEY
        default_model: claude-sonnet-5-5
      fallback:
        type: openai
        api_key: env://OPENAI_API_KEY
        default_model: gpt-5
      local:
        type: ollama
        base_url: http://127.0.0.1:11434/v1
        default_model: qwen3:8b
    
    agents:
      - name: resilient
        version: 1.0.0
        system_instructions: Produce a concise, verified response.
        default_model:
          name: quality
          provider: primary
          model: claude-sonnet-5-5
          quality_tier: 3
          context_window_tokens: 32000
          context_safety_margin_tokens: 512
        fallback_models:
          - name: cloud-fallback
            provider: fallback
            model: gpt-5
            quality_tier: 2
            context_window_tokens: 16000
            context_safety_margin_tokens: 512
          - name: local-fallback
            provider: local
            model: "qwen3:8b"
            quality_tier: 1
            context_window_tokens: 12000
            context_safety_margin_tokens: 512
        budget:
          max_output_tokens: 4096
    

    A few things are worth noticing before we run anything.

    Keys under providers: are instance names, not vendors. primary, fallback and local are the names the agent refers to; type: picks the adapter. You can run two Ollama instances, or two Azure deployments, and each gets its own health tracking, rate limit, circuit breaker and cost accounting.

    Secrets are references. env://ANTHROPIC_API_KEY is resolved just before a call, and a literal key in the file is rejected when the settings load.

    Typos fail at load time. If a fallback names a provider that isn't configured, load_settings() raises a ConfigurationError naming the exact path. A misspelt fallback never sits unnoticed until the day you need it.

    Context windows are declared, not guessed. The runtime doesn't infer a window from a model name. The context_window_tokens values here are deployment choices; set them to what your deployment actually supports.

    Run it with real keys and you'll see which route answered:

    pip install -e '.[dev]'
    export ANTHROPIC_API_KEY=...   # primary
    export OPENAI_API_KEY=...      # cloud fallback
    python -m examples.pattern_provider_fallback.app "What year did the first moon landing occur?"
    
    [Route: anthropic/claude-sonnet-5-5]
    The first moon landing occurred in 1969, when Apollo 11 landed on the lunar surface on July 20th.
    

    Or, with no keys at all, run the same router with a primary that fails on purpose:

    python -m examples.pattern_provider_fallback.app --offline
    
    [offline: the primary provider fails on purpose; the router falls back to mock]
    [Route: mock/deterministic]
    deterministic fallback response
    

    One call, three candidates

    Here is what happens inside one model call when the primary is down.

    An agent run sends one model call to the router. The router tries the primary, claude-sonnet-5-5 on Anthropic, which returns 503 and is charged one failure. It then tries fallback 1, gpt-5 on OpenAI, which answers. Fallback 2, qwen3:8b on Ollama, is not tried. The run records model.completed with provider openai and model gpt-5.

    The router builds a list of candidates from the default model and the fallbacks, drops any that can't serve the request, then tries them in turn until one answers. Two things decide whether a failure moves on to the next candidate or ends the call.

    Which failures fall back

    The question the router asks of every error is: would another provider fail the same way?

    The provider saysFalls back?Why
    5xx, transport error, overloadedYesThis provider is having a bad time
    Timeout, including HTTP 408YesSame
    429 rate limitYesYour quota there is used up, not everywhere
    401 / 403YesA rotated or revoked key is specific to that provider
    404 for a model it doesn't serveYesAnother provider may serve its own model
    Invalid or unparseable responseYesThat model misbehaved
    400 / 422 invalid requestNoEvery provider would reject the same request

    The last row is the important one. A 400 usually means your request is broken: a tool schema the API rejects, an unsupported parameter, a malformed message. Falling back would get you an answer from the next model, possibly a worse one, and the bug would ship. So the router raises the error as it is, and doesn't count it against the provider's health either, because the provider did nothing wrong.

    One case looks like a provider failure but isn't: a missing API key. The key is resolved before the call, so a missing ANTHROPIC_API_KEY is a configuration error and stops the run rather than quietly routing everything to the fallback for weeks.

    Break it on purpose

    The router is plain Python, so you can drive it directly with a fake provider and watch every decision. This script uses the runtime's real ModelRouter and its mock provider. Save it anywhere and run it from a checkout of the runtime:

    import asyncio
    import time
    
    from algen_agent_runtime.exceptions.errors import BudgetExceededError, ProviderError
    from algen_agent_runtime.models.base import ModelRouter
    from algen_agent_runtime.models.providers.mock import MockModelProvider
    from algen_agent_runtime.types.contracts import (
        ErrorKind, Message, ModelCapabilities, ModelProfile, ModelRequest, Role,
    )
    
    
    class FlakyProvider:
        """A provider that fails however we tell it to; failure=None means healthy."""
    
        def __init__(self, provider_id: str) -> None:
            self.provider_id = provider_id
            self.failure: ProviderError | None = None
            self.calls = 0
    
        async def capabilities(self, model: str) -> ModelCapabilities:
            return ModelCapabilities(chat=True)
    
        async def generate(self, request: ModelRequest):
            self.calls += 1
            if self.failure:
                raise self.failure
            response = await MockModelProvider(responses=("ok",)).generate(request)
            return response.model_copy(update={"provider": self.provider_id})
    
        async def health(self) -> bool:
            return self.failure is None
    

    Three failures to throw, in the shape a real adapter raises them:

    UNAVAILABLE = ProviderError("503 overloaded", ErrorKind.UNAVAILABLE, True, status_code=503)
    BAD_REQUEST = ProviderError("400 bad tool schema", ErrorKind.INVALID_REQUEST, status_code=400)
    NOT_SERVED = ProviderError("404 no such model", ErrorKind.INVALID_REQUEST, status_code=404)
    

    A primary with a 32k window and a fallback with 16k, tried in declaration order. The circuit breaker is tightened to three failures and a two-second reset so the demo runs quickly; the defaults are five failures and 30 seconds.

    PROFILES = (
        ModelProfile(name="quality", provider="primary", model="big",
                     routing_preference="ordered", context_window_tokens=32_000),
        ModelProfile(name="cloud-fallback", provider="fallback", model="mid",
                     routing_preference="ordered", context_window_tokens=16_000),
    )
    
    primary, fallback = FlakyProvider("primary"), FlakyProvider("fallback")
    router = ModelRouter(circuit_failure_threshold=3, circuit_reset_seconds=2)
    router.register_provider(primary)
    router.register_provider(fallback)
    
    
    async def call(label: str, failure: ProviderError | None, text: str = "hi") -> None:
        primary.failure = failure
        before = primary.calls
        request = ModelRequest(messages=(Message.text(Role.USER, text),), max_output_tokens=1024)
        try:
            response = await router.generate(request, PROFILES)
            result = f"-> {response.provider}/{response.model}"
        except (ProviderError, BudgetExceededError) as exc:
            result = "raised: " + str(exc).split("; ")[-1]
        tried = "tried" if primary.calls > before else "skipped"
        print(f"{label:<16} primary {tried:<8} {result}")
    
    
    async def main() -> None:
        for n in range(1, 6):
            await call(f"{n}. 503", UNAVAILABLE)  # the circuit opens after the third
        await call("6. recovered", None)  # still open: not tried
        time.sleep(2.1)
        await call("7. after reset", UNAVAILABLE)  # tried once, fails, reopens at once
        time.sleep(2.1)
        await call("8. after reset", None)  # tried, succeeds, circuit closes
        await call("9. 400 request", BAD_REQUEST)
        await call("10. 404 model", NOT_SERVED)
        await call("11. 20k prompt", UNAVAILABLE, "log line " * 9_000)
    
    
    asyncio.run(main())
    

    The output, from runtime 0.1.0a23:

    1. 503           primary tried    -> fallback/mid
    2. 503           primary tried    -> fallback/mid
    3. 503           primary tried    -> fallback/mid
    4. 503           primary skipped  -> fallback/mid
    5. 503           primary skipped  -> fallback/mid
    6. recovered     primary skipped  -> fallback/mid
    7. after reset   primary tried    -> fallback/mid
    8. after reset   primary tried    -> primary/big
    9. 400 request   primary tried    raised: 400 bad tool schema
    10. 404 model    primary tried    -> fallback/mid
    11. 20k prompt   primary tried    raised: fallback/mid: configured context budget exceeded
    

    Every line is a decision worth understanding. Let's go through them in groups.

    Calls 1 to 8: the circuit breaker

    The first three calls try the primary, get a 503 and fall back. That's correct but slow: during an outage, every request pays for a doomed call to the primary before it reaches a model that works. If the primary's failure is a timeout, that's the whole timeout, every time.

    So after a run of consecutive failures, the primary's circuit opens, and calls 4 to 6 don't call it at all. Call 6 is the interesting one: the primary has actually recovered, but the router doesn't know that and doesn't check. It keeps sending traffic to the fallback until the reset period has passed.

    Eight calls with a failure threshold of three. Calls one to three try the primary, get 503 and the fallback answers; the circuit opens after the third. Calls four to six skip the primary, even after it recovers. After a pause, call seven tries the primary, which still fails, and the circuit reopens at once. After another pause, call eight tries the primary, it answers, and the circuit closes.

    After the reset (call 7), the primary is a candidate again, but on probation. Its failure count wasn't cleared, so a single further failure reopens the circuit straight away rather than allowing another three. Only a success (call 8) closes the circuit and resets the count.

    A few properties fall out of how this is built:

    • Health is per provider instance. Two Azure deployments of the same model have separate breakers, so one bad region doesn't take out the other.
    • A malformed tool call isn't an outage. If the model answers but writes a tool call that doesn't parse, the router moves on without charging the provider's health; the provider worked, its model just made a mistake.
    • A rejected request isn't an outage either. Calls that fail with a 400 or 422 never count towards opening the circuit.

    Call 9: the request that fails everywhere

    The primary rejects the request with a 400, and the router raises it. The fallback is never called. Your logs show the real error, 400 bad tool schema, instead of a successful answer from a different model with your bug still in place.

    Call 10: the 404 that doesn't

    A 404 looks like a client error, but here it means "I don't serve that model", which says nothing about whether the fallback serves its model. So it falls back. This catches a real class of incident: a provider retires a model version, or a deployment name in one region is wrong.

    The distinction is made on both the error kind and the status code. An invalid_request with status 400 or 422 (or no status at all) stops; anything else that's provider-specific moves on.

    Call 11: when the fallback is too small

    The last call sends a prompt of roughly 20k tokens while the primary is down. The fallback has a 16k window, so the router has a choice: send it anyway and let the provider reject it, cut the conversation until it fits, or refuse.

    It refuses, and it does so for a precise reason. Before every dispatch, the router computes how much room there is for messages on that candidate:

    message allowance = context window
                      - requested output tokens
                      - safety margin
                      - estimated tool definitions, response schema and extensions
    

    It then packs the request into that allowance. The system message, the latest user message and the newest group of messages are protected. Tool calls stay together with their results. Older groups can be dropped, oldest first. The output reservation is never silently shrunk to make room.

    The same request, a 1k system message, five 1k history groups and a 9k latest message, is packed separately for each candidate with a 4k output reserve and a safety margin. The 32k primary fits everything. The 16k fallback keeps the system message, the newest history group and the latest message, and drops the four oldest groups. The 12k local fallback can't fit even the system and latest messages beside the reserve, so it is skipped without a call.

    Each candidate is packed from the original request, not from what the previous candidate received. A small primary can't strip history that a larger fallback would have kept.

    If the protected part doesn't fit, the candidate is skipped before anything is spent: no rate-limit slot, no metered credits, no call and no mark against its health. In call 11 the primary failed and the fallback couldn't fit, so the router raised an error naming both reasons. That's the right outcome. An agent answering a 20k-token question with the question cut off would be worse than an error.

    When the primary is healthy, the same 20k prompt goes to the 32k primary as normal. Budgets decide where a request can go; they don't change where it goes first.

    The estimate is based on the size of the serialised content, not a provider's tokenizer, which is why the safety margin exists. If you see "context budget exceeded" for prompts that should fit, the declared window is usually smaller than the deployment really supports.

    Ordered or balanced

    The demo used routing_preference: ordered, which tries candidates strictly in the order you declared them. Providers with an open circuit, missing capabilities or exceeded latency or cost limits are still skipped.

    default_model:
      name: primary
      provider: gemini
      model: gemini-flash-latest
      routing_preference: ordered
    fallback_models:
      - name: fast
        provider: groq
        model: openai/gpt-oss-120b
        routing_preference: ordered
    

    ordered describes the whole list, so configuration that sets it on only some of an agent's models is rejected when the settings load.

    The default, balanced, ranks candidates by a score instead: 1,000 points per quality_tier, minus the provider's last observed latency in milliseconds, minus its average cost per 1k tokens. One tier is worth a full second of latency. That means a primary that's up but slow can lose its place: at tier 3 and 3.5 seconds it scores −500, while a tier 2 fallback answering in 1.2 seconds scores 800 and is tried first.

    That can be exactly what you want for an interactive agent, and exactly what you don't want when the fallback is a cheaper, weaker model. Pick deliberately:

    • ordered when the fallbacks are there for outages only and quality must not drift.
    • balanced when the candidates are roughly interchangeable and latency matters most.
    • local_first / cloud_first when locality matters more than anything else, for example keeping data on a local model whenever one is healthy.

    You can also make a candidate ineligible outright: required_capabilities (such as tools or json_schema) removes models that can't do the job, and max_latency_ms or max_cost_per_1k_tokens removes ones that are too slow or too expensive.

    Retries sit outside fallback

    Fallback and retries are separate layers, and it helps to know which does what.

    The router works through the candidate list once per attempt. If every candidate fails and all of those failures are retryable, the runtime retries the whole chain with exponential backoff, according to the agent's retry_policy:

    agents:
      - name: resilient
        retry_policy:
          max_attempts: 3              # the default
          initial_backoff_seconds: 0.1
          max_backoff_seconds: 5
    

    Each retry is recorded as a model.retrying event with the attempt number, the error kind and the delay. A failure that isn't retryable, such as the 400 in call 9, ends the attempt without a retry.

    Streaming is stricter. Before a streamed call starts, the router can skip a candidate whose window is too small for the request. After that it's committed: a provider error doesn't fall back, and it never switches providers once tokens have started arriving. Replaying half an answer from a different model would be worse than failing.

    Know which model answered

    Fallback that you can't see is a quality problem waiting to be found. Every model call in a run emits model.completed with the provider and model that actually answered, along with tokens, latency and estimated cost. The example reads it back like this:

    completed = [
        event.data
        for event in await container.runtime.events.history(result.run_id)
        if event.type == "model.completed"
    ]
    route = f"{completed[-1]['provider']}/{completed[-1]['model']}" if completed else None
    

    Send these events to your tracing (Traccia, or any OpenTelemetry backend) and you can answer "how much of yesterday's traffic ran on the fallback?" with a query rather than a guess. Because metering happens per call, the credit metering for a fallback call is charged against the model that actually ran, at its own price.

    Bring your own provider

    Any provider can join a chain, as long as it reports failures in the router's terms. Implement the ModelProvider protocol and translate the vendor's errors into a ProviderError with an ErrorKind, an honest retryable flag and the HTTP status. A sketch for an HTTP-based adapter:

    except httpx.HTTPStatusError as exc:
        status = exc.response.status_code
        kind = {
            401: ErrorKind.AUTHENTICATION,
            403: ErrorKind.AUTHORIZATION,
            408: ErrorKind.TIMEOUT,
            429: ErrorKind.RATE_LIMIT,
        }.get(status, ErrorKind.UNAVAILABLE if status >= 500 else ErrorKind.INVALID_REQUEST)
        raise ProviderError(
            f"{self.provider_id} returned {status}",
            kind,
            retryable=status in {408, 429} or status >= 500,
            status_code=status,
        ) from exc
    

    Getting this mapping right is most of the work. Report a provider outage as INVALID_REQUEST with status 400 and the router will never fall back from it; report a broken request as UNAVAILABLE and every bug becomes a silent fallback. OpenAI-compatible endpoints can reuse OpenAICompatibleProvider, which already does the translation, and register with type: openai_compatible and a base_url.

    What this doesn't solve

    The circuit breaker is per process. Each runtime process keeps its own provider health in memory. Ten replicas each learn about an outage separately, and each pays its own failed calls before its circuit opens.

    The breaker settings aren't in agent.yaml yet. The threshold (5) and reset period (30 seconds) are fixed when the service builds its router. Code that constructs ModelRouter directly can change them, as the demo does.

    There's no single trial request after a reset. When the reset period ends, every call in flight may try the primary at once, not just one. If it's still down, they all fail over, and the circuit reopens.

    A fallback is a different model. The runtime makes sure the fallback can do the job (capabilities, context window, cost and latency limits) and records that it ran. It can't make it answer as well as the primary. Evaluate your agent on each model in its chain, not just the first.

    The token estimate is approximate, and configured budgets currently support text messages only; an image or file block with a budget set is a configuration error.

    It's early. At the time of writing the runtime is 0.1.0a23, a pre-release.

    Five things worth copying

    • Fall back on the provider's failure, never on the request's. A 400 that falls back turns a bug into a quality regression nobody notices.
    • Stop calling what's down. A circuit breaker turns an outage from "every request is slow" into "every request goes straight to the fallback".
    • Pack for each model separately, protect the user's latest message, and skip a model rather than cut the question to fit.
    • Make the fallback order a decision. Strict order for outage-only fallbacks; a score only when the models are truly interchangeable.
    • Record which model answered, every time, next to what it cost.

    An outage at your model provider is going to happen. With a declared chain, a breaker and an honest record of who answered, it becomes a dip on a dashboard rather than an incident.