Collect Feedback

Overview

Confident AI allows you to collect feedback from end users that are interacting with your LLM app. A thumbs up/down after a chatbot reply, a star rating at the end of a support conversation, a "was this helpful?" prompt — all of these are signals about how your AI is actually performing in production, and they're often the earliest warning you'll get that something regressed. End user feedback can be left on:

• Traces
• Spans, and
• Threads

When you send an annotation of a user feedback, you'll get the opportunity to incorporate them into a dataset, so the conversations your users flagged become the test cases you evaluate against next.

How It Works

To collect feedback, you need to:

• Setup a custom UI for users to enter their rating (thumbs up/down or 5 star system), and optionally expected outcome/output, and explanation
• Either collect the trace UUID, span UUID, or thread ID you'd like to leave feedback for
• Send the feedback to Confident AI via the Confident AI SDK or the Confident API

Since the thread ID is something you provide (click here if unsure) during LLM tracing, it is generally easier to setup feedback collection on threads than on traces and spans.

Collect Single-Turn Feedback

1. Get the OpenTelemetry identifiers

   Read the trace and span IDs while your application span is active, and store them alongside the response so your feedback UI can refer back to the correct request later.

   PythonTypescript

   from openai import OpenAI
   from confident_trace import init, span, shutdown
   from opentelemetry import trace
   
   init()
   client = OpenAI()
   
   def llm_app(query: str):
       with span("llm_app", type="agent"):
           res = client.chat.completions.create(
               model="gpt-4o",
               messages=[{"role": "user", "content": query}]
           ).choices[0].message.content
   
           current_span = trace.get_current_span()
           context = current_span.get_span_context()
           trace_id = f"{context.trace_id:032x}" if context.is_valid else None
           span_id = f"{context.span_id:016x}" if context.is_valid else None
           return res, trace_id, span_id
   
   try:
       output, TRACE_ID, SPAN_ID = llm_app("Write me a poem.")
   finally:
       shutdown()

2. Send annotation for trace/span

   In a separate workflow — typically the handler behind your thumbs up/down button — send the feedback with the Confident AI SDK using the OpenTelemetry IDs you collected. The annotation API still names these fields trace_uuid and span_uuid.

   For a trace:

   PythonTypeScriptcURL

   from confident_ai import ConfidentAI
   from confident_ai.annotations import TraceAnnotationInput
   
   client = ConfidentAI()
   
   result = client.annotations.create(
       annotation=TraceAnnotationInput(
           trace_uuid="3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
           value="<ANNOTATION-VALUE-INPUT>"
       ),
   )

   For a span:

   PythonTypeScriptcURL

   from confident_ai import ConfidentAI
   from confident_ai.annotations import SpanAnnotationInput
   
   client = ConfidentAI()
   
   result = client.annotations.create(
       annotation=SpanAnnotationInput(
           span_uuid="9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f",
           value="<ANNOTATION-VALUE-INPUT>"
       ),
   )

   Set value to the user's rating. A thumbs rating, the default, takes true for thumbs up and false for thumbs down. For a 5 star rating, set field_type to FIVE_STAR_RATING and value to a whole number from 1 to 5. The expected_output your user wanted and their explanation are optional.

Collect Multi-Turn Feedback

1. Setup thread ID

   Define a thread ID and configure your traced LLM app to associate all related traces to this thread. Set it on the trace inside your application span — update_trace / updateTrace writes to the outermost span of the current trace, so it works from anywhere in the request.

   PythonTypescript

   from openai import OpenAI
   from confident_trace import init, span, update_trace, shutdown
   
   init()
   THREAD_ID = "YOUR-THREAD-ID"
   client = OpenAI()
   
   def llm_app(query: str) -> str:
       with span("llm_app", type="agent"):
           response = client.chat.completions.create(
               model="gpt-4o",
               messages=[{"role": "user", "content": query}]
           ).choices[0].message.content
           update_trace(thread_id=THREAD_ID, input=query, output=response)
           return response
   
   try:
       llm_app("Write me a poem.")
   finally:
       shutdown()

2. Send annotation for thread

   Post the thread-level feedback with the Confident AI SDK using the thread IDs you defined. Because the thread ID is yours, there's nothing to look up — the only thing to keep in mind is that the traces for that thread need to have been ingested first, which usually means waiting a few seconds after the response is sent.

   PythonTypeScriptcURL

   from confident_ai import ConfidentAI
   from confident_ai.annotations import ThreadAnnotationInput
   
   client = ConfidentAI()
   
   result = client.annotations.create(
       annotation=ThreadAnnotationInput(
           thread_id="thread-42",
           value="<ANNOTATION-VALUE-INPUT>"
       ),
   )

   As with single-turn feedback, value is true or false for a thumbs rating, or a whole number from 1 to 5 with field_type set to FIVE_STAR_RATING. The expected_outcome your user wanted and their explanation are optional.

Next Steps

Once feedback is flowing in, put it to work:

Last updated on October 19, 2026