LiteLLM | Confident AI Docs

Overview

LiteLLM is a tool that makes it easy for developers to connect to and use many different large language models (LLMs)—like those from OpenAI, Anthropic, Hugging Face, and more—through a single, simple interface.

Tracing Quickstart

Confident AI integrates with LiteLLM to trace your LLM calls for both Proxy and SDK in the Observatory.

LiteLLM Proxy is a server (or gateway) that acts as a central point for accessing many different language models through a single API endpoint whereas the SDK is a client library that allows you to use LiteLLM in your own application.

Setup via SDK

Install the LiteLLM SDK:

Install Dependencies

$ pip install litellm

Configure LiteLLM

Use litellm.success_callback and litellm.failure_callback to trace your LLM calls:

showLineNumbers

1 import os
2 import time
3 import litellm
4 
5 os.environ['OPENAI_API_KEY']='<your-openai-api-key>'
6 os.environ['CONFIDENT_API_KEY']='<your-confident-api-key>'
7 
8 litellm.success_callback = ["deepeval"]
9 litellm.failure_callback = ["deepeval"]
10 
11 response = litellm.completion(
12     model="gpt-3.5-turbo",
13     messages=[
14         {"role": "user", "content": "What's the weather like in San Francisco?"}
15     ],
16 )

Setup via Proxy

Configure LiteLLM

Add deepeval to the success_callback and failure_callback in the config.yaml file.

config.yaml

1 model_list:
2   - model_name: gpt-4o
3     litellm_params:
4       model: gpt-4o
5 litellm_settings:
6   success_callback: ["deepeval"]
7   failure_callback: ["deepeval"]

Set environment variables

Add your OpenAI and Confident API keys in .env file.

$ OPENAI_API_KEY=<your-openai-api-key>
> CONFIDENT_API_KEY=<your-confident-api-key>

Start server

Run the following command to start the proxy server:

$ litellm --config config.yaml --debug

Make a request

Run the following command to make a request to the proxy server:

$ curl -X POST 'http://0.0.0.0:4000/chat/completions' \
> -H 'Content-Type: application/json' \
> -H 'Authorization: Bearer sk-1234' \
> -d '{
>     "model": "gpt-3.5-turbo",
>     "messages": [
>       {
>         "role": "system",
>         "content": "You are a helpful math tutor. Guide the user through the solution step by step."
>       },
>       {
>         "role": "user",
>         "content": "how can I solve 8x + 7 = -23"
>       }
>     ]
> }'

There are two ways to start the LiteLLM proxy server from CLI i.e., either using pip package or docker container. We have used the pip package in the example above. Refer to the LiteLLM documentation for more information.