Crew AI | Confident AI Docs

Overview

CrewAI is a lean, lightning-fast Python framework for creating autonomous AI agents tailored to any scenario. Confident AI allows you to trace and evaluate CrewAI workflows with just a single line of code.

Tracing Quickstart

Try in Google Colab

Install Dependencies

Run the following command to install the required packages:

$ pip install -U deepeval crewai

Configure CrewAI

Instrument CrewAI with your Confident AI API key using instrument_crewai.

main.py

1 from crewai import Task, Crew, Agent
2 
3 from deepeval.integrations.crewai import instrument_crewai
4 instrument_crewai()
5 
6 agent = Agent(
7     role="Consultant",
8     goal="Write clear, concise explanation.",
9     backstory="An expert consultant with a keen eye for software trends.",
10 )
11 
12 task = Task(
13     description="Explain the given topic",
14     expected_output="A clear and concise explanation.",
15     agent=agent,
16 )
17 
18 crew = Crew(agents=[agent], tasks=[task])
19 
20 result = crew.kickoff({"input": "What are the LLMs?"})

Run CrewAI

Kickoff your crew by executing the script:

$ python main.py

You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

Advanced Usage

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Set the thread_id in the trace context and call crew.kickoff within the context.

main.py

1 ...
2 with trace(thread_id="crewai_run_1"):
3     crew.kickoff({"city": "London"})

Logging metadata

You can also set the metadata in the trace context.

main.py

1 ...
2 with trace(metadata={"test_metadata_1": "test_metadata_1"}):
3     crew.kickoff({"city": "London"})

Other trace attributes

Additionally, you can set the name, tags and user_id in the trace context.

main.py

1 ...
2 with trace(name="crewai_run_1", tags=["crewai"], user_id="crewai_user_1"):
3     crew.kickoff({"city": "London"})

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your OpenAI Agent, which will run evaluations on all incoming traces on Confident AI’s servers. This is the recommended approach, especially if your agent is in production.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your OpenAI Agent.

Click to see supported metrics for OpenAI Agents

Confident AI supports evaluating the input-output pairs of OpenAI Agent spans and traces, which means your metric collections must only contain metrics that only require the input and output for evaluation. These metrics include:

If you’re looking to use other metrics, setup Confident AI’s native tracing instead.

Create metric collection

Run evals

Run evaluations on the various components of your CrewAI application by setting the metric_collection to the DeepEval’s wrapper for CrewAI.

The current CrewAI integration supports metrics with parameters that evaluate input and actual output in addition to the Task Completion metric.

Trace

Crew Span

Agent Span

LLM Span

To evaluate the trace level, set the trace_metric_collection to the DeepEval’s trace context.

main.py

1 from crewai import Task, Crew, Agent
2 
3 from deepeval.tracing import trace
4 from deepeval.integrations.crewai import instrument_crewai
5 instrument_crewai()
6 
7 agent = Agent(
8     role="Consultant",
9     goal="Write clear, concise explanation.",
10     backstory="An expert consultant with a keen eye for software trends.",
11 )
12 
13 task = Task(
14     description="Explain the given topic",
15     expected_output="A clear and concise explanation.",
16     agent=agent,
17 )
18 
19 crew = Crew(agents=[agent], tasks=[task])
20 
21 with trace(trace_metric_collection="test_collection_1"):
22     result = crew.kickoff({"input": "What are the LLMs?"})

All incoming traces and spans will now be evaluated using metrics from your metric collection.