Collect Feedback

Overview

Confident AI allows you to collect feedback from end users that are interacting with your LLM app. A thumbs up/down after a chatbot reply, a star rating at the end of a support conversation, a "was this helpful?" prompt — all of these are signals about how your AI is actually performing in production, and they're often the earliest warning you'll get that something regressed. End user feedback can be left on:

• Traces
• Spans, and
• Threads

When you send an annotation of a user feedback, you'll get the opportunity to incorporate them into a dataset, so the conversations your users flagged become the test cases you evaluate against next.

How It Works

To collect feedback, you need to:

• Setup a custom UI for users to enter their rating (thumbs up/down or 5 star system), and optionally expected outcome/output, and explanation
• Either collect the trace UUID, span UUID, or thread ID you'd like to leave feedback for
• Send the feedback to Confident AI via the Evals API

Since the thread ID is something you provide (click here if unsure) during LLM tracing, it is generally easier to setup feedback collection on threads than on traces and spans.

Collect Single-Turn Feedback

1. Get the OpenTelemetry identifiers

   Read the trace and span IDs while your application span is active, and store them alongside the response so your feedback UI can refer back to the correct request later.

   PythonTypescript

   from openai import OpenAI
   from confident_trace import init, span, shutdown
   from opentelemetry import trace
   
   init()
   client = OpenAI()
   
   def llm_app(query: str):
       with span("llm_app", type="agent"):
           res = client.chat.completions.create(
               model="gpt-4o",
               messages=[{"role": "user", "content": query}]
           ).choices[0].message.content
   
           current_span = trace.get_current_span()
           context = current_span.get_span_context()
           trace_id = f"{context.trace_id:032x}" if context.is_valid else None
           span_id = f"{context.span_id:016x}" if context.is_valid else None
           return res, trace_id, span_id
   
   try:
       output, TRACE_ID, SPAN_ID = llm_app("Write me a poem.")
   finally:
       shutdown()

2. Send annotation for trace/span

   In a separate workflow — typically the handler behind your thumbs up/down button — send the feedback with DeepEval using the OpenTelemetry IDs you collected. The annotation API still names these fields trace_uuid and span_uuid.

   PythonTypescript

   from deepeval.annotation import send_annotation
   
   send_annotation(
       trace_uuid=TRACE_ID,
       rating=1,
       # span_uuid=SPAN_ID, # you can only set trace_uuid or span_uuid
   )
   

   from deepeval.annotation.api import AnnotationType
   from deepeval.annotation import send_annotation
   
   send_annotation(
       trace_uuid=TRACE_ID,
       type=AnnotationType.FIVE_STAR_RATING,
       rating=5
       # span_uuid=SPAN_ID, # you can only set trace_uuid or span_uuid
   )

Collect Multi-Turn Feedback

1. Setup thread ID

   Define a thread ID and configure your traced LLM app to associate all related traces to this thread. Set it on the trace inside your application span — update_trace / updateTrace writes to the outermost span of the current trace, so it works from anywhere in the request.

   PythonTypescript

   from openai import OpenAI
   from confident_trace import init, span, update_trace, shutdown
   
   init()
   THREAD_ID = "YOUR-THREAD-ID"
   client = OpenAI()
   
   def llm_app(query: str) -> str:
       with span("llm_app", type="agent"):
           response = client.chat.completions.create(
               model="gpt-4o",
               messages=[{"role": "user", "content": query}]
           ).choices[0].message.content
           update_trace(thread_id=THREAD_ID, input=query, output=response)
           return response
   
   try:
       llm_app("Write me a poem.")
   finally:
       shutdown()

2. Send annotation for thread

   Post the thread-level feedback to the Evals API using the thread IDs you defined. Because the thread ID is yours, there's nothing to look up — the only thing to keep in mind is that the traces for that thread need to have been ingested first, which usually means waiting a few seconds after the response is sent.

   PythonTypescript

   from deepeval.annotation import send_annotation
   
   send_annotation(
       thread_id=THREAD_ID,
       rating=1,
   )
   

   from deepeval.annotation.api import AnnotationType
   from deepeval.annotation import send_annotation
   
   send_annotation(
       thread_id=THREAD_ID,
       type=AnnotationType.FIVE_STAR_RATING,
       rating=5
   )

Next Steps

Once feedback is flowing in, put it to work:

Last updated on September 10, 2026