Failover & Routing: Keep Model Calls Up When a Provider Isn't

Every provider has bad hours

Rate limits, timeouts, quota errors, and 5xx spikes hit every model provider. Without a plan, each one becomes a failed request in your product, or a retry loop in your code that nobody budgeted for.

• Hand-rolled retry loops in every service, each with different rules
• Failover to another provider happens in code, outside your guardrails and budgets
• A provider switch in the middle of a conversation throws away its prompt cache
• Retries multiply spend with no cap on what a single request can cost

Retry, then fail over

When a call fails, Cloptima retries it. If the provider still cannot serve it, the call moves to another provider that serves the same model. Conversations stay where their prompt cache is warm, so failover does not throw your caching away.

• Automatic retries on every request
• Automatic failover to another provider that serves the same model
• Conversations stay where their prompt cache is warm
• Optional backup models for the rare model that cannot be served
• The same budgets and guardrails for every retry and failover

Nothing to set up

Send your requests through the gateway and failover is already on. Ask for a model by name, and Cloptima can serve it from any of your providers that offer it. When you want a safety net beyond that model, add backup models to the policy.

Same policy layer as budgets and guardrails

Every request passes your policy, guardrails, and budget before it runs, and retries and failover follow the same rules. Spend is priced and attributed to the provider that served each call.