Your entire AI feature has a single point of failure, and it's not your code. It's the inference endpoint you call. One provider, one model, one network path between your product and the thing that makes it work. When that endpoint rate-limits you during a traffic spike, your feature rate-limits with it. When it returns a 529 because it's overloaded, your users get errors. When it slows down, every request in your product that touches the model slows down. You didn't build a resilient system; you built a thin wrapper around someone else's availability, and you inherited their worst minute as your worst minute.
This isn't hypothetical and it isn't rare. Providers have capacity events. Models get deprecated on a schedule you don't control. Latency varies with their load, not yours. A system that assumes the inference call always succeeds, always fast, against the same model forever, is a system that has confused a happy-path demo for production architecture.
Provider abstraction first
You cannot route, fall back, or swap models if every line of code that calls a model is welded to one provider's SDK and one model string. The prerequisite for everything else is an abstraction layer: your application code calls your interface — "complete this," "embed this" — and the layer underneath handles which provider, which model, which credentials, which retry policy.
This is plumbing, and it's the plumbing that makes the rest possible. Done right, swapping a model is a config change, not a refactor. Adding a second provider is implementing one interface, not touching every call site. Without it, the model choice is hard-coded in a hundred places and every change is a migration. The abstraction is cheap to build early and expensive to retrofit once the SDK calls have metastasized through the codebase.
The cost to watch: don't abstract so hard you lose provider-specific features. Prompt caching, adaptive thinking, structured outputs, the effort dial — these are real levers that move cost and quality, and a lowest-common-denominator interface that hides them is throwing away the reason you picked a capable model. Abstract the call shape; expose the features that matter through the interface, not around it.
Route by task and by cost
Once you can reach multiple models through one interface, routing becomes a lever instead of an accident. Different requests want different models, and sending everything to your single most capable model is paying frontier prices to classify support tickets.
Route by task. A frontier model at $5 input / $25 output per million tokens earns its cost on hard reasoning and long-horizon work. It's overkill for intent classification, simple extraction, or a short rewrite — those run fine on a smaller, faster, cheaper model. A capable mid-tier at $3/$15 and a fast tier at $1/$5 cover a large share of real traffic. Map each task to the cheapest model that does it well, decided up front, predictable for cost forecasting.
Route by cost-quality tradeoff per surface. Latency-sensitive, low-stakes paths take the fast model. Quality-critical paths take the capable one. The routing table is an architectural decision you make deliberately and tune with data — measured per-route quality and cost, not guessed. The default of "everything goes to the best model" is a decision too, just an expensive one nobody made on purpose.
Fallback chains for when the primary fails
Routing picks the best model for a healthy request. Fallback decides what happens when that model is unavailable, and that's the difference between a feature that degrades gracefully and one that errors out.
Define an ordered chain per route. Primary model. On failure — rate limit, overload, timeout, error — fall to the secondary. The secondary might be the same model on a different provider, or a different model that's good enough to keep the feature alive. The principle is graceful degradation: a slightly less capable answer beats an error page, for most features. A request that would have errored instead completes on the backup, and the user never knows the primary was down.
Distinguish retryable from terminal. A 429 (rate limit) and a 529 (overloaded) are retryable — back off and retry, then fall over if it persists. The SDKs retry these automatically with exponential backoff, which handles the transient blip; the fallback chain handles the sustained outage the retries can't ride out. A 400 (bad request) is terminal — retrying or falling over won't help, because the request itself is malformed, and burning the fallback chain on a permanent error just wastes time and money. Classify the failure, then act on the class.
Match the fallback to the route's stakes. A best-effort summary can fall all the way to a cheap model. A high-stakes generation might fall to another capable model and no further, failing loudly rather than quietly serving a worse answer where quality is the point. Not every route should degrade the same distance.
Circuit breakers so failure doesn't cascade
Retrying a dead provider on every request turns its outage into your outage, with extra latency. When a provider is clearly down, stop hammering it.
A circuit breaker tracks failures against a model or provider. Past a threshold, it opens — requests skip that provider entirely and go straight to the fallback, no wasted timeout, no retry storm. After a cooldown it half-opens, tests with a trickle of traffic, and closes again when the provider recovers. This is the pattern that keeps a provider's bad ten minutes from becoming your bad hour. Without it, every request pays the full timeout against a dead endpoint before falling over, and your latency collapses across the board while you politely retry something that isn't answering.
The breaker also gives you a clean signal. An open circuit is an alert: this provider is down, the fallback is carrying load, someone should know. You learn about the outage from a ./automate health monitor, not from the latency graph cratering and a wave of support tickets.
Eval-gated model swaps
Multi-model architecture makes swapping models easy. Easy is dangerous — a swap that's trivial to deploy is trivial to deploy wrong. Gate every model change on your evals before it touches production traffic.
A new model — a provider's upgrade, a cheaper option, a different tier for a route — runs against your eval set first, and the scores get compared against the incumbent. Better or equal, ship it. Worse, don't, and understand why before you try again. The subtlety that catches teams: a newer model can be genuinely more capable and still score lower on your evals, because it follows instructions more literally, calibrates length differently, or escapes JSON another way that breaks your parser. That's a harness mismatch, not a regression — but you only know which it is because you gated on your eval set instead of the provider's benchmark. Never swap a production model on the strength of someone else's published numbers. Gate on yours, every time.
This is where routing, fallback, and evaluation meet. The abstraction lets you swap. The evals tell you whether the swap is safe. Together they turn "we're locked to this model and terrified to touch it" into "we move between models deliberately, with proof."
What fixed looks like
Your application calls one interface, and the provider, model, and retry policy live behind it. Requests route to the cheapest model that does each task well, decided deliberately and tuned with measured data. Every route has an ordered fallback chain, so a provider outage degrades gracefully instead of erroring. Failures are classified retryable versus terminal and handled accordingly. Circuit breakers cut off dead providers before they cascade, and an open circuit pages someone. Model swaps are gated on your eval set, never a stranger's benchmark. When a provider has a bad ten minutes, your users have a normal ten minutes.
This is for you if
You're a funded US company with an AI feature in production, real traffic, and a single provider whose every hiccup becomes your outage. Resilience architecture of this kind is part of an AI build or audit, typically $50k+; designing provider abstraction, routing, fallback, and eval-gated swaps into a larger product runs $100k+.
It's not for you if you're pre-launch with no traffic — provider outages don't hurt a feature nobody uses yet, and premature abstraction is complexity you'll pay for before you need it. It's not for you if your feature is genuinely tolerant of downtime and an occasional error costs nothing; build the fallback where reliability is worth its weight. And it's not for you if you want multi-provider as a checkbox without the eval harness to swap safely — routing you can't validate is just more ways to ship a regression.