Throttling & Retries for AI Connections
Tune request concurrency, timeouts, and retries for your AI connection.
Overview
These settings control how Confident AI sends requests to your AI Connection endpoint—how long to wait for a response, how many requests to send at once, and how many times to retry on failure. Tune them to keep large evaluation runs from overwhelming your AI app while staying resilient to transient errors.
Request Timeout
Set the maximum time (in seconds) that Confident AI will wait for your endpoint to respond before timing out. This helps prevent evaluations from hanging indefinitely if your AI connection is slow or unresponsive.
- Minimum: 1 second
- Default: 60 seconds
Max Concurrency
Set the maximum number of concurrent requests that Confident AI will send to your endpoint at the same time. This helps prevent overwhelming your AI app during large evaluation runs.
- Minimum: 1
- Default: 20
Max Retries
Set the maximum number of times Confident AI will retry a failed request to your endpoint. This helps handle transient errors without failing the entire evaluation.
- Minimum: 0
- Default: 0
Next Steps
With throttling and retries dialed in, secure your endpoint and link results back to traces.
Authorization
Secure requests with a secrets manager and Auth0 or HMAC authentication.
Linking Traces
Link test cases and turns to their traces for full observability.
Last updated on