LiteLLM

Overview

LiteLLM is a tool that makes it easy for developers to connect to and use many different large language models (LLMs)—like those from OpenAI, Anthropic, Hugging Face, and more—through a single, simple interface.

Tracing Quickstart

Confident AI integrates with LiteLLM to trace your LLM calls for both Proxy and SDK in the Observatory.

Setup via SDK

Install the LiteLLM SDK:

1. Install Dependencies

   pip install litellm

2. Configure LiteLLM

   Use litellm.success_callback and litellm.failure_callback to trace your LLM calls:

   import os
   import time
   import litellm
   
   os.environ['OPENAI_API_KEY']='<your-openai-api-key>'
   os.environ['CONFIDENT_API_KEY']='<your-confident-api-key>'
   
   litellm.success_callback = ["deepeval"]
   litellm.failure_callback = ["deepeval"]
   
   response = litellm.completion(
       model="gpt-3.5-turbo",
       messages=[
           {"role": "user", "content": "What's the weather like in San Francisco?"}
       ],
   )

Setup via Proxy

1. Configure LiteLLM

   Add deepeval to the success_callback and failure_callback in the config.yaml file.

   model_list:
     - model_name: gpt-4o
       litellm_params:
         model: gpt-4o
   litellm_settings:
     success_callback: ["deepeval"]
     failure_callback: ["deepeval"]

2. Set environment variables

   Add your OpenAI and Confident API keys in .env file.

   OPENAI_API_KEY=<your-openai-api-key>
   CONFIDENT_API_KEY=<your-confident-api-key>

3. Start server

   Run the following command to start the proxy server:

   litellm --config config.yaml --debug

4. Make a request

   Run the following command to make a request to the proxy server:

   curl -X POST 'http://0.0.0.0:4000/chat/completions' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer sk-1234' \
   -d '{
       "model": "gpt-3.5-turbo",
       "messages": [
         {
           "role": "system",
           "content": "You are a helpful math tutor. Guide the user through the solution step by step."
         },
         {
           "role": "user",
           "content": "how can I solve 8x + 7 = -23"
         }
       ]
   }'