Portkey

Overview

Confident AI lets you trace and evaluate Portkey LLM calls, whether standalone or used as a component within a larger application.

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval portkey-ai

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")

   export CONFIDENT_API_KEY="<your-confident-api-key>"

3. Configure Portkey

   To begin tracing your Portkey LLM calls as a component in your application, import OpenAI and use the PORTKEY_GATEWAY_URL to trace the calls.

   Chat CompletionsAsync Chat CompletionsResponsesAsync Responses

   from deepeval.openai import OpenAI
   from portkey_ai import PORTKEY_GATEWAY_URL
   
   portkey = OpenAI(
   base_url = PORTKEY_GATEWAY_URL,
   api_key = "<PORTKEY_API_KEY>"
   )
   
   response = portkey.chat.completions.create(
       model = "@slug/<model>",
       messages = [
       {"role": "system", "content": "You are a helpful assistant."},
       {"role": "user", "content": "What is Portkey"}
       ],
   )

4. Run Portkey

   Invoke your agent by executing the script:

   python main.py

   You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

Advanced Usage

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt to the create method.

from portkey_ai import PORTKEY_GATEWAY_URL

from deepeval.openai import OpenAI
from deepeval.prompt import Prompt
from deepeval.tracing import trace

portkey = OpenAI(
  base_url = PORTKEY_GATEWAY_URL,
  api_key = "<PORTKEY_API_KEY>"
)

prompt = Prompt(alias="my_prompt")
prompt.pull(version="00.00.01")

with trace(prompt=prompt):
  response = portkey.chat.completions.create(
      model = "@slug/<model>",
      messages = [
        {"role": "system", "content": prompt.interpolate(name="John")}, # string system prompt
        {"role": "user", "content": "What is Portkey"}
      ],
  )

print(response.choices[0].message.content)

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. Learn more about threads here. You can set the thread_id in the trace context.

from deepeval.openai import OpenAI
from deepeval.tracing import trace

from portkey_ai import PORTKEY_GATEWAY_URL

portkey = OpenAI(
  base_url = PORTKEY_GATEWAY_URL,
  api_key = "<PORTKEY_API_KEY>"
)

with trace(thread_id="test_thread_id_1"):
  response = portkey.chat.completions.create(
      model = "@slug/<model>",
      messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Portkey"}
      ],
  )

print(response.choices[0].message.content)

Evals Usage

Online evals

If your OpenAI application is in production, and you still want to run evaluations on your traces, use online evals. It lets you run evaluations on all incoming traces on Confident AI's server.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your OpenAI agent. Copy the name of the metric collection.

2. Run evals

   Set the llm_metric_collection name in the trace context when invoking your OpenAI client to evaluate Llm Spans.

   from deepeval.openai import OpenAI
   from deepeval.tracing import trace
   
   client = OpenAI()
   
   with trace(llm_metric_collection="test_collection_1"):
       response = client.chat.completions.create(
           model="gpt-4o",
           messages=[
               {"role": "system", "content": "You are a helpful assistant."},
               {"role": "user", "content": "Hello, how are you?"},
           ],
       )

End-to-end evals

Confident AI allows you to run end-to-end evals on your OpenAI client to evaluate your Portkey calls directly. This is recommended if you are testing your Portkey calls in isolation.

1. Create metric

   from deepeval.metrics import AnswerRelevancyMetric
   
   task_completion = AnswerRelevancyMetric(
       threshold=0.7,
       model="gpt-4o-mini",
       include_reason=True
   )

2. Run evals

   Replace your OpenAI client with DeepEval's. Then, use the dataset's evals_iterator to invoke your OpenAI client for each golden. Remember to replace base_url and api_key with the Portkey gateway URL and API key.

   Chat CompletionsResponsesAsync Chat CompletionsAsync Responses

   from deepeval.openai import OpenAI
   from deepeval.metrics import AnswerRelevancyMetric, BiasMetric
   from deepeval.dataset import EvaluationDataset
   from deepeval.tracing import trace
   
   client = OpenAI(
       base_url = PORTKEY_GATEWAY_URL,
       api_key = "<PORTKEY_API_KEY>"
   )
   
   dataset = EvaluationDataset()
   dataset.pull("your-dataset-alias")
   
   for golden in dataset.evals_iterator():
       with trace(
           llm_metrics=[AnswerRelevancyMetric(), BiasMetric()],
           expected_output=golden.expected_output,
       ):
           client.chat.completions.create(
               model="gpt-4o",
               messages=[
                   {"role": "system", "content": "You are a helpful assistant."},
                   {"role": "user", "content": golden.input}
               ],
           )

Using OpenAI in component-level evals

You can also evaluate Portkey calls through component-level evals. This approach is recommended if you are testing your Portkey calls as a component in a larger application system.

1. Create metric

   from deepeval.metrics import AnswerRelevancyMetric
   
   task_completion = AnswerRelevancyMetric(
       threshold=0.7,
       model="gpt-4o-mini",
       include_reason=True
   )

2. Run evals

   Replace your OpenAI client with DeepEval's. Then, use the dataset's evals_iterator to invoke your LLM application for each golden.

   Chat CompletionsResponsesAsync Chat CompletionsAsync Responses

   from deepeval.openai import OpenAI
   from deepeval.tracing import observe, trace
   from deepeval.dataset import EvaluationDataset
   from deepeval.metrics import AnswerRelevancyMetric
   
   client = OpenAI(
       base_url = PORTKEY_GATEWAY_URL,
       api_key = "<PORTKEY_API_KEY>"
   )
   
   @observe()
   def generate_response(input: str) -> str:
       with trace(
           llm_metrics=[AnswerRelevancyMetric()],
           expected_output=golden.output,
       ):
           response = client.chat.completions.create(
               model="gpt-4.1",
               messages=[
                   {"role": "system", "content": "You are a helpful assistant."},
                   {"role": "user", "content": input},
               ],
           )
           return response
   
   # Create dataset
   dataset = EvaluationDataset()
   dataset.pull("your-dataset-alias")
   
   # Run component-level evaluation
   for golden in dataset.evals_iterator():
       generate_response(golden.input)

Last updated on July 27, 2026