Confident AI - The DeepEval LLM Evaluation Platform

USE CASES

Build your AI moat.
Do evals the right way.

Confident AI provides an opinionated solution to curate dataset, align metrics, and automate LLM testing with tracing. Teams use it to safeguard AI systems to save hundreds of hours a week on fixing breaking changes, cut inference cost by 80%, and convince stakeholders that their AI is always better than the week before.

READ CASE STUDY TRY IT NOW

END-TO-END EVALUATION

REGRESSION TESTING

DEEPEVAL AND PLATFORM

Built for developers.
Used by everyone to drive product decisions.

Easily integrate evals using DeepEval, with intuitive product analytic dashboards for non-technical team members.

Testing reports Tracing observability Dataset editor Prompt management

observability.py

1
2
3
4
5
6
7
8
9
10
11
from deepeval.tracing import observe
  
@observe()  
def llm_app(input: str):
res = openai.ChatCompletion.create(
         model="gpt-4o",
         messages=[{"role": "user", "content": input}],
       ).choices[0].message["content"]
    return res
  
llm_app("Write me a poem.")

evaluate.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from deepeval.tracing import observe, update_current_span
from deepeval.dataset import Golden, LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate
  
@observe(metrics=[AnswerRelevancyMetric()])  # Define metrics
def llm_app(input: str):
res = openai.ChatCompletion.create(model="gpt-4o", messages=[{"role": "user", "content": input}])
       .choices[0].message["content"]
  
   # Set test case at runtime 
 update_current_span(test_case=LLMTestCase(input=input, actual_output=res))
    return res
  
# Call your LLM app to evaluate 
evaluate(goldens=[Golden(input="Write me a poem."), observable_callback=llm_app)

dataset.py

1
2
3
4
5
6
7
8
9
10
11
from deepeval.dataset import EvaluationDataset, Golden
  
# Pull dataset for evaluation
dataset = EvaluationDataset()
dataset.pull(alias="My Testset")
print(dataset.goldens)
  
# Or, push goldens to update dataset
new_goldens = [Golden(input="Write me a poem.")]
dataset = EvaluationDataset(goldens=new_goldens)
dataset.push(alias="My Testset", overwrite=False)

prompt.py

1
2
3
4
5
6
7
8
9
10
from deepeval.prompt import Prompt
  
# Pull prompt from cloud 
prompt = Prompt(alias="System Prompt")
prompt.pull(version="00.00.18")
messages_to_llm = prompt.interpolate(variable="value")
  
#  Pass interpolated prompt to LLM
res = openai.ChatCompletion.create(model="gpt-4o", messages=messages_to_llm).choices[0].message["content"]
print(res)