Get Started

Introduction

Evaluate LLM applications of all sorts, with and without code.

What is Confident AI?

Confident AI is the end-to-end platform for teams to quality-assure their AI applications — from RAG pipelines and agentic workflows to chatbots and core LLM models.

LLM evaluation allows engineers, QAs, and PMs to:

  • Prevent regressions - Catch breaking changes before they reach production
  • Optimize performance - Find the best prompts, models, and parameters for your use case
  • Build confidence - Get data-driven insights into your AI application’s quality
  • Save time - Automate manual testing with 40+ pre-built evaluation metrics
  • Enable iteration - Compare different versions of your AI system objectively
  • Quality assurance - Ensure consistent performance across different inputs and scenarios
Confident AI's evals are 100% powered by DeepEval

DeepEval is one of the most widely adopted LLM evaluation framework in the world, with over 10k stars and 20 million daily evaluations.

Star History Chart

⭐ DeepEval Star Growth ⭐

While DeepEval is like Pytest for LLM apps, Confident AI is the dashboard UI for DeepEval.

How LLM evals work

Evaluation on Confident AI has three core components:

LLM Metrics

Automates manual pre-deployment AI testing with 40+ LLM-as-a-Judge metrics.

Dataset Curation

Annotate or generate test datasets prepared by QAs and PMs.

LLM Tracing

Real-time AI execution observability with performance tracking.

You don’t strictly need to have a dataset coming into Confident AI to start running evals, as you can setup tracing and run evals on the fly instead.

Confident AI supports evals and tracing for any LLM use case, including multi-turn ones.

Key capabilities

  • Comprehensive single-turn, multi-turn LLM testing
  • Experiment with different versions of prompts and models
  • Detect unexpected breaking changes through evals
  • LLM tracing to debug and monitor in production
  • Track product analytics and user stats
  • Include human-in-the-loop to notice what needs to be worked on

Choose your quickstart

LLM Evals in Development

Best for: Those that have a dataset or are already doing manual testing


  • Create and annotate a dataset
  • Unit-test for LLM app regressions
  • Find the best prompts, models, etc.
  • Learn how to run evals in directly in the UI

Setup your LLM eval pipeline for pre-deployment quality assurance

LLM Tracing for Production Evals

Best for: Running ad-hoc evals immediately and building datasets from traced data


  • Setup LLM observability to trace AI executions
  • Enable online metrics to run ad-hoc evals in production
  • Learn how to run offline evals on historical traces

Perfect for those without a dataset to begin with

FAQs

While DeepEval computes the metric results required for data-driven LLM app development, it does not provide the insights required for iteration.

Click here for a more comprehensive comparison.

All types of LLM use cases are supported, including summarization, Text-SQL, custom support chatbots, internal RAG QAs, conversational agents, etc.

These use cases can be of any system, including RAG pipelines, agentic workflows, conversational chatbots, or just a combination of everything (e.g. RAG chatbots, agentic RAG).

Confident AI has tailored metrics and platform capabilities for different types of LLM applications, and it is extremely important to adjust your evaluation strategy depending on your LLM use case. You can read more on the different types of use cases on this page.

Complex agentic systems are definitely supported on Confident AI through LLM tracing. One thing to note though is that it is extremely important to decide carefully on what to (not) evaluate in a complex LLM agentic workflow, since trying to evaluate everything means you’re actually evaluating nothing.

Yes, Confident AI offers SSO, data segregation for teams, user roles and permissions (with customizations available), and well as the ability to self-host in your cloud premises.

We’re proudly HIPAA compliant and are willing to sign BAAs with customers on the Premium subscription plan or above.

Yes, while most users are using the SaaS offering, your organization can deploy Confident AI in your cloud premises (e.g. AWS, Azure, GCP, etc.) through a dockerized manner, which includes integrations with your existing identify providers (e.g. Azure AD, Ping, Okta, etc.) of choice for authentication into Confident AI platform. In our experience, this process takes 1-2 weeks max.

No credit card upfront is required, and we offer transparent pricing with 4 different tiers which includes a generous free tier. You can view the full pricing here.

We try to make it sure that you only pay for something once you have had the chance to try it out. If you don’t think this is the case, please email support@confident-ai.com and we will make things more generous.