Multi-Turn State

Overview

During multi-turn simulations, Confident AI calls your AI Connection endpoint once per turn. You can use state to persist information—like a thread ID or session—across turns so your AI app can maintain context throughout the conversation.

On the first turn, the state variable in your payload will be empty since no prior state exists. If your endpoint returns a state object and the state key path successfully extracts it, that state will be included in the state payload variable from the second turn onwards.

Payload

To enable multiturn state, include state in your payload configuration so it gets sent to your endpoint on each turn:

1
{
2
  "input": golden.input,
3
  "state": state
4
}

Here’s an example of how your endpoint might handle state:

1
@app.post("/generate")
2
def generate(request: dict):
3
    state = request.get("state", {})
4
5
    if not state:
6
        thread_id = create_new_thread()
7
    else:
8
        thread_id = state["threadId"]
9
10
    response = llm.generate(
11
        thread_id=thread_id,
12
        user=request["input"]
13
    )
14
15
    return {
16
        "output": response,
17
        "state": {"threadId": thread_id}
18
    }

State Key Path

The state key path works just like the actual output key path—a list of strings or integers representing the path to the state object in your JSON response. This tells Confident AI where to extract state from your endpoint’s response so it can be passed back on the next turn.

For example, if your endpoint returns:

1
{
2
  "output": "Hello! How can I help?",
3
  "state": {
4
    "threadId": "abc-123"
5
  }
6
}

Set the state key path to ["state"].

Next Steps

Now that your AI connection can maintain context across turns, link each turn back to its trace for full observability.