Human-in-the-Loop

Leave Annotations on UI

Learn about annotation queues, and how to assign items for team members to annotate

Overview

Confident AI allows internal, domain experts to leave annotations on traces, spans, and threads in addition to automatic ingestion of user feedback via the Evals API.

You can either leave annotations as an:

  • Ad-hoc standealone task, or
  • As part of an annotation queue

Both of the two workflow gives you the same end-result.

“Annotation queues” refer to a group of traces, spans, or threads that are pending to be evaluated. It provides an extra layer of abstraction to manage annotations from different team members more effectively.

Annotate as a Standalone Task

Leaving annotations as a standealone task is extremely simple and only requires you to navigate to either the the Traces, Spans, or Threads page under the Observatory.

Each and every single trace/span/thread you click on will give you the ability to leave scores in the form of:

  • A thumbs up/down, or
  • A 1-5 star rating

You’ll also have the opportunity to leave optional fields such as:

  • Explanation
  • Expected output (traces and spans)
  • Expected outcome (threads)

To understand the difference in traces/spans and thread annotations, read this section on single vs multi-turn annotations.

Using Annotation Queues

An often preferred way, especially for larger teams that require annotations from domain experts, is to use annotation queues for annotation.

Annotation queues are basically a group of traces/spans/threads, that are pending to be evaluated. It provides annotation teams a more organized and streamlined interface to annotate data instead of the ad-hoc approach shown above.

There are three types of annotation queues in Confident AI: Traces, Spans, and Threads. This mean that you cannot add threads to an annotation queue that are meant for traces, and vice-versa.

1

Create annotation queue

First create an annotation queue. In this entire example, we’ll be showing an annotation queue for traces, but it will be almost identical for spans and threads.

You must click on the Traces tab to create a queue for traces. The same applies for Spans and Threads.

Create Annotation Queue
2

Add items to queue

You can add traces, spans, and threads to an annotation queue literally whenever you see one of them on the platform. This mainly includes the Observatory for Traces, Spans, and Threads, but also component-level testing reports where traces and spans are displayed.

You can add to multiple queues at once, and even assign a team member to annotate the items you’re queueing.

If you don’t see any avaiable queues to add to, make sure you’ve created a queue specific to the data you’re adding (e.g. trace queue for traces, span queue for spans, and thread queue for threads).

Add Traces to Queues
3

Annotate queued items

After you’ve added items into your annotation queue, they will be visible in your annotation queue for annotation:

Annotate Queued Traces

You’ll have the option to:

  • Track completion progress
  • Filter for items that are completed, still in progress, or assigned to you
  • Auto-mark items as completed when done with annotation
  • View full details of traces/spans/threads

By default, the Queue Annotator strips away all information that except the input, output, metadata, and turns (for threads).

4

Track progress

Once you’re done, go to Queue Settings to see an overview of all completed/in progress items.

Manage Queued Items

For items that you’ve queued for annotation, you can always manage them via the Queue Settings page, which includes assigning/unnassigning users, marking items as completed/in progress, and removing items from a queue.

Manage Queue Settings