When Should I Start Tracing?

In the third week of Confident AI's trial period, a number of teams come to us really disappointed, having not scoped out a set of metrics that stakeholders believe and can rely on. We try to help them by logging into their dashboard, only to figure out that there's no traces to look at. They tell us why they couldn't set up tracing because of other priorities, we show them it takes less than 10 minutes to set up, they get convinced and walk away feeling really bad.

We have written this article specifically so that you don't run into the same mistakes.

Advantages of tracing early

There are two strong advantages to setting up tracing as early as possible:

• You can get signals from traces automatically. Once you start getting the data in, you don't really even need to run evaluations. AI can do the job for you.
• You can always run annotations retrospectively in a nice UI, but querying a database just kills the mood for any non-technical SME or PM who wants to help out and run annotations.

All of this contributes to writing better metrics, and you can always do that in less than 30 minutes if you have all the data readily available on the platform.

Four reasons why engineers convince themselves that it's not the right time for tracing

1. Our AI app isn't ready. It's still being built, and we're just running ad hoc right now.
2. We should only trace when we start getting real traffic. The traces we have right now in development aren't useful at all.
3. We don't have the metrics in place, so why would we start tracing? We already have data for something else.
4. Setting up tracing will take too much of my time. I am occupied with building the app right now, not doing other stuff.

Why should you resist the urge not to start tracing immediately?

Out of everything, tracing gives you the lowest barrier to value, and the highest signal to failure. Your app may not be ready, but the truth is, how long could integration actually take when we already offer you more than 20 integrations? Furthermore, you can just plug something into Claude or Cursor nowadays, and they'll build an entire tracing setup for you in under five minutes.

What's important is seeing your AI evolve over time. Evolution can mean moving from development to staging, beta to alpha, or even from beta and out of beta into production.

So what matters is that you trace first. Signals on Confident AI will immediately surface to you if there are any problems that we have identified. If the signals are legitimate, you can look at them. If not, you can ignore them. But the point is that when it comes to deployment, it should feel like a natural progression of seeing how your AI is evolving over time. It's meaningful progress that also allows the platform and recommend the correct metrics. By merely having traces on the platform, we unlock signals, insights, and metrics altogether without anyone spending any time on annotations at all.