Crew AI | Confident AI Docs

Overview

CrewAI is a lean, lightning-fast Python framework for creating autonomous AI agents tailored to any scenario. Confident AI allows you to trace and evaluate CrewAI workflows with just a single line of code.

Tracing Quickstart

Try in Google Colab

Install Dependencies

Run the following command to install the required packages:

$ pip install -U deepeval crewai

Setup Confident AI Key

$ export CONFIDENT_API_KEY="<your-confident-api-key>"

Configure CrewAI

Instrument CrewAI with instrument_crewai before running any crew. You only need to call this once at startup.

main.py

1 from crewai import Task, Crew, Agent
2 
3 from deepeval.integrations.crewai import instrument_crewai
4 instrument_crewai()
5 
6 agent = Agent(
7     role="Consultant",
8     goal="Write clear, concise explanation.",
9     backstory="An expert consultant with a keen eye for software trends.",
10 )
11 
12 task = Task(
13     description="Explain the given topic",
14     expected_output="A clear and concise explanation.",
15     agent=agent,
16 )
17 
18 crew = Crew(agents=[agent], tasks=[task])
19 
20 result = crew.kickoff({"input": "What are the LLMs?"})

instrument_crewai() monkey-patches Crew.kickoff, Crew.kickoff_for_each, their async variants (kickoff_async, kickoff_for_each_async, akickoff, akickoff_for_each), and Agent.execute_task / Agent.aexecute_task. It also registers an event listener that captures LLM calls, tool usage, and knowledge retrieval automatically.

Run CrewAI

Kickoff your crew by executing the script:

$ python main.py

You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

What gets traced

After calling instrument_crewai(), every crew run produces a nested trace with the following span types:

Span type	What it captures
Crew	The top-level `Crew.kickoff` (or async variant) call — input, output, and timing for the whole crew run
Agent	Each `Agent.execute_task` call — the task prompt sent to the agent and the agent’s final answer
LLM	Every LLM call made by an agent — messages sent, model name, provider, and the model response
Tool	Every tool invocation — tool name, input arguments, output, and timing
Knowledge Retrieval	Knowledge retrieval operations — the query and the retrieved documents

Advanced Usage

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Set the thread_id in the trace context and call crew.kickoff within the context.

main.py

1 from deepeval.tracing import trace
2 
3 ...
4 with trace(thread_id="crewai_run_1"):
5     crew.kickoff({"city": "London"})

Logging metadata

You can also set the metadata in the trace context.

main.py

1 from deepeval.tracing import trace
2 
3 ...
4 with trace(metadata={"test_metadata_1": "test_metadata_1"}):
5     crew.kickoff({"city": "London"})

Other trace attributes

Additionally, you can set the name, tags and user_id in the trace context.

main.py

1 from deepeval.tracing import trace
2 
3 ...
4 with trace(name="crewai_run_1", tags=["crewai"], user_id="crewai_user_1"):
5     crew.kickoff({"city": "London"})

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your CrewAI application, which will run evaluations on all incoming traces on Confident AI’s servers. This is the recommended approach, especially if your agent is in production.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your CrewAI application.

Click to see supported metrics for CrewAI

Confident AI supports evaluating the input-output pairs of CrewAI spans and traces, which means your metric collections must only contain metrics that only require the input and output for evaluation. These metrics include:

If you’re looking to use other metrics, setup Confident AI’s native tracing instead.

Create metric collection

Run evals

Run evaluations on the various components of your CrewAI application by setting the metric_collection on DeepEval’s wrappers for Crew, Agent, LLM, or tool.

The current CrewAI integration supports metrics with parameters that evaluate input and actual output in addition to the Task Completion metric.

Trace

Crew Span

Agent Span

LLM Span

Tool Span

To evaluate at the trace level, pass metric_collection to the trace context.

main.py

1 from crewai import Task, Crew, Agent
2 
3 from deepeval.tracing import trace
4 from deepeval.integrations.crewai import instrument_crewai
5 instrument_crewai()
6 
7 agent = Agent(
8     role="Consultant",
9     goal="Write clear, concise explanation.",
10     backstory="An expert consultant with a keen eye for software trends.",
11 )
12 
13 task = Task(
14     description="Explain the given topic",
15     expected_output="A clear and concise explanation.",
16     agent=agent,
17 )
18 
19 crew = Crew(agents=[agent], tasks=[task])
20 
21 with trace(metric_collection="test_collection_1"):
22     result = crew.kickoff({"input": "What are the LLMs?"})

All incoming traces and spans will now be evaluated using metrics from your metric collection.