Crew AI

Use Confident AI for LLM observability and evals for CrewAI

Overview

CrewAI is a lean, lightning-fast Python framework for creating autonomous AI agents tailored to any scenario. Confident AI allows you to trace and evaluate CrewAI workflows with just a single line of code.

Tracing Quickstart

Try in Google Colab

Install Dependencies
Run the following command to install the required packages:
```
pip install -U deepeval crewai
```

Setup Confident AI Key

export CONFIDENT_API_KEY="<your-confident-api-key>"

deepeval login

import deepeval

deepeval.login("<your-confident-api-key>")

Configure CrewAI

Instrument CrewAI with instrument_crewai before running any crew. You only need to call this once at startup.

main.py

from crewai import Task, Crew, Agent

from deepeval.integrations.crewai import instrument_crewai
instrument_crewai()

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff({"input": "What are the LLMs?"})

Run CrewAI
Kickoff your crew by executing the script:
```
python main.py
```

You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

What gets traced

After calling instrument_crewai(), every crew run produces a nested trace with the following span types:

Span type	What it captures
Crew	The top-level `Crew.kickoff` (or async variant) call — input, output, and timing for the whole crew run
Agent	Each `Agent.execute_task` call — the task prompt sent to the agent and the agent's final answer
LLM	Every LLM call made by an agent — messages sent, model name, provider, and the model response
Tool	Every tool invocation — tool name, input arguments, output, and timing
Knowledge Retrieval	Knowledge retrieval operations — the query and the retrieved documents

Advanced Usage

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Set the thread_id in the trace context and call crew.kickoff within the context.

main.py

from deepeval.tracing import trace

...
with trace(thread_id="crewai_run_1"):
    crew.kickoff({"city": "London"})

Logging metadata

You can also set the metadata in the trace context.

main.py

from deepeval.tracing import trace

...
with trace(metadata={"test_metadata_1": "test_metadata_1"}):
    crew.kickoff({"city": "London"})

Other trace attributes

Additionally, you can set the name, tags and user_id in the trace context.

main.py

from deepeval.tracing import trace

...
with trace(name="crewai_run_1", tags=["crewai"], user_id="crewai_user_1"):
    crew.kickoff({"city": "London"})

View Trace Attributes

namestr

The name of the trace. Learn more.

tagsList[str]

Tags are string labels that help you group related traces. Learn more.

metadataDict

Attach any metadata to the trace. Learn more.

thread_idstr

Supply the thread or conversation ID to view and evaluate conversations. Learn more.

user_idstr

Supply the user ID to enable user analytics. Learn more.

Evals Usage

Online evals

You can run online evals on your CrewAI application, which will run evaluations on all incoming traces on Confident AI's servers. This is the recommended approach, especially if your agent is in production.

Create metric collection
Create a metric collection on Confident AI with the metrics you wish to use to evaluate your CrewAI application.
Click to see supported metrics for CrewAI
Confident AI supports evaluating the input-output pairs of CrewAI spans and traces, which means your metric collections must only contain metrics that only require the input and output for evaluation. These metrics include:
If you're looking to use other metrics, setup Confident AI's native tracing instead.
Create metric collection

Run evals

Run evaluations on the various components of your CrewAI application by setting the metric_collection on DeepEval's wrappers for Crew, Agent, LLM, or tool.

To evaluate at the trace level, pass metric_collection to the trace context.

main.py

from crewai import Task, Crew, Agent

from deepeval.tracing import trace
from deepeval.integrations.crewai import instrument_crewai
instrument_crewai()

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])

with trace(metric_collection="test_collection_1"):
    result = crew.kickoff({"input": "What are the LLMs?"})

To evaluate at the crew span level, replace Crew with DeepEval's wrapper and set metric_collection.

main.py

from crewai import Task, Agent

from deepeval.integrations.crewai import Crew, instrument_crewai

instrument_crewai()

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task], metric_collection="test_collection_1")

result = crew.kickoff({"input": "What are the LLMs?"})

To evaluate at the agent span level, replace Agent with DeepEval's wrapper and set metric_collection.

main.py

from crewai import Task, Crew

from deepeval.integrations.crewai import Agent, instrument_crewai

instrument_crewai()

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
    metric_collection="test_collection_1",
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff({"input": "What are the LLMs?"})

To evaluate at the LLM span level, replace LLM with DeepEval's wrapper and set metric_collection.

main.py

from crewai import Task, Crew, Agent

from deepeval.integrations.crewai import LLM, instrument_crewai

instrument_crewai()

llm = LLM(
    model="gpt-4o-mini",
    metric_collection="test_collection_1",
)

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
    llm=llm,
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff({"input": "What are the LLMs?"})

To evaluate at the tool span level, replace the @tool decorator with DeepEval's wrapper and set metric_collection.

main.py

from crewai import Task, Crew, Agent

from deepeval.integrations.crewai import tool, instrument_crewai

instrument_crewai()

@tool(metric_collection="test_collection_1")
def search_database(query: str) -> str:
    """Search the internal database for relevant information."""
    return f"Results for: {query}"

agent = Agent(
    role="Consultant",
    goal="Write clear, concise explanation.",
    backstory="An expert consultant with a keen eye for software trends.",
    tools=[search_database],
)

task = Task(
    description="Explain the given topic",
    expected_output="A clear and concise explanation.",
    agent=agent,
)

crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff({"input": "What are the LLMs?"})

Need help wiring this into your stack?Bring traces and evals into the tools your team already usesTalk to an expert

Overview

Tracing Quickstart

Install Dependencies

Setup Confident AI Key

Configure CrewAI

Run CrewAI

What gets traced

Advanced Usage

Logging threads

Logging metadata

Other trace attributes

Evals Usage

Online evals

Create metric collection

Run evals