On this page
01
What you'll learn
Providers have bad hours: timeouts, rate limits, and server errors. Cloptima handles them on every request, with nothing to configure. In about seven minutes you will see what happens when a call fails, and add a safety net for the rare model that fails everywhere.
- Automatic retries for short failures
- Automatic failover to another provider that serves the same model
- Conversations that keep their prompt cache
- Backup models, as an optional safety net
- Budgets and guardrails that apply to retries and failover
Retries and failover need no setup. To add backup models, you need an owner or admin role and a policy bound to your traffic.
02
Retry, then fail over
When a call fails, Cloptima retries it. If the provider still cannot serve it, the request moves to another provider that serves the same model.
1Call fails
Timeout, rate limit, server error
2Retry
Short failures often clear
3Fail over
Another provider that serves the model
4Answer returned
Your code does not change
03
Ask by model name for the widest failover
When your requests name a model without a provider, Cloptima can fail over across every provider that serves it.
- Failover uses only providers you can already call
- Your policy lists decide which providers are allowed
The guide on asking for a model by name shows how providers are chosen and ordered.
04
Conversations keep their prompt cache
A provider keeps a prompt cache for the account that wrote it. Cloptima keeps each conversation where its cache is warm, so long conversations stay cheap.
Failover moves a conversation only when its provider cannot serve it.
05
Add backup models
Backup models are a safety net for the rare case when the requested model cannot be served. They are optional.
- 1
Open the policy
Go to AI → Policies and edit the policy that governs the traffic.
- 2
Find Backup models
It sits under Allowed models and Denied models.
- 3
Pick your backup models
They are tried in the order you pick them.
- 4
Save
Save the policy.
- Used only when the requested model cannot be served
- Later requests still ask for the requested model first
- The same budgets and guardrails apply to backup models
06
Budgets apply to every retry
Retries, failovers, and backup models follow the same budgets, credits, and guardrails as the first call. You pay for what providers bill.
07
Roll it out
Failover is already on. These steps give it more room to work.
- 1
Ask for models by name
Failover then reaches the providers that serve the model.
- 2
Add a second provider
Add a credential for another provider that serves your busiest model, or run on Cloptima credits.
- 3
Add backup models
Only where an answer matters more than which model gives it.
- 4
Watch Provider spend
A sudden shift in the Explorer's Provider grouping means failover is happening.
08
If something goes wrong
Most surprises come from credentials or from a model that is down everywhere.
| What you see | Likely cause | Fix |
|---|---|---|
| Requests still fail when a provider is down | No second provider you can use | Add a credential for another provider that serves the model |
| Spend shifts to a pricier provider | Failover is working during an incident | Check the provider's status, then compare Provider spend |
| Spend shows on a backup model | The requested model could not be served | Check the model's providers, or add one that serves it |
| Backup models are refused on save | Too many, a repeat, or a model the catalog does not offer | Pick different models from the list |