Project Settings

Setup LLM Connection

Reach your LLM app through the internet via a RESTFUL LLM connection

Overview

You can setup an LLM endpoint that accepts a POST request over HTTPS to enable users to run evaluations directly on the platform without having to code, and start an evaluation through a click of a button instead.

Understanding LLM Connection

At a high level, you would have to provide Confident AI with the mappings to test case parameters such as the actual_output, retrieval_context, etc.

At evaluation time, Confident AI will use goldens defined in your dataset to prompt your LLM connection. Therefore, it is important that you LLM connection is setup to be ready to accept golden parameters such as the input (for single-turn evals), or turns (for multi-turn evals). Keep reading to find out more.

How to Setup an LLM Connection

Create an LLM Endpoint

In order for Confident AI to reach your LLM application, you’ll need to expose your LLM in a RESTFUL API endpoint that is accessible over the internet. These are the hard rules you MUST follow when setting up your endpoint:

  1. Accepts a POST request over HTTPS.
  2. Returns a JSON response and MUST contain the actual_output value somewhere in the returned JSON object. Whether or not to supply a retrieval_context or tools_called value in your returned JSON is optional, and this depends on whether the metrics you have enabled for your experiment requires these parameters.

Parse Request Data

When Confident AI calls your LLM endpoint, it makes a POST request with two different data structures.

We highly recommend that your LLM endpoint accepts both single and multi-turn request data to not run into unexpected errors during evaluation.

Single-turn evals

For single-turn evals, this would be the request data:

1{   
2    "input": "...",
3    "context": ["...", "..."], // Optional, might not be sent if not in golden
4    "prompts": [{"alias": "...", "version": "..."}] // Optional, might not be sent if prompts aren't configured
5}

The input and context will correspond to the values in your goldens. Since context is an optional field, your LLM connection should be flexible in parsing it. You can also configure prompts in the LLM connection settings, which will be included in the request data as prompts, a list of objects with the prompt alias and version.

Any JSON response returned will be parsed and used to deduce what the remaining test case parameters (i.e. actual_output) values are. So, it is imperative that your LLM endpoint:

  1. Parses the incoming data to extract this input value to carry out generations.
  2. Returns the actual_output and any other LLMTestCase parameters in the JSON response with their correct respective type.

Multi-turn evals

For multi-turn evals, this would be the request data:

1{   
2    "turns": [{"role": "user", "content": "..."}],
3    "context": ["...", "..."], // Optional, might not be sent if not in golden
4    "prompts": [{"alias": "...", "version": "..."}] // Optional, might not be sent if prompts aren't configured
5}

The turns data represents the turn history, and your LLM app behind this custom LLM connection should take it into account in its generation. If your conversational goldens have 3 existing turns, they will be sent here in this turns array.

If you’re running simulations on the platform, Confident AI will also send over the simulated turns in the turns array. In other words, we manage the state of the turns, and all your app has to do is take it into account during generation.

Connect Your LLM Endpoint

You can setup your LLM connection in Project Settings > LLM Connection. There is also a button for you to ping your LLM endpoint to sanity check that you’ve setup everything correctly.

You’ll have to provide:

  1. The HTTPS endpoint you’ve setup.
  2. The JSON key path to the mandatory actual_output parameter, and the optional retrieval_context , and tools_called parameters. The JSON key path is a list of strings.

In order for evaluation to work, you MUST set the JSON key path for the actual_output parameter. Remember, the actual_output of your LLM application is always required for evaluation, while the retrieval_context and tools_called parameters are optional depending on the metrics you’ve enabled.

The JSON key path tells Confident AI where to look in your JSON response for the respective test case parameter values.

For instance, if you set the key path of the actual_output parameter to ["response", "actual output"], the correct JSON response to return from your LLM endpoint is as follows:

1{
2  "response": {
3    "actual output": "must be a string!"
4  }
5}

That’s not to say you can’t include other things to return in your JSON response, but the key path will determine the variables Confident AI will be using to populate LLMTestCases at evaluation time.

If you’re wondering why expected_output, context, and expected_tools is not required in setting up your JSON key path, it is because it is expected that these variables are static, just like the input, and should therefore come from your dataset instead.

Once you’re done, click on the Ping Endpoint button, and see if you’re getting a 200 status response. If not, fix the problem based on the error on the screen.

Configure Prompts

To configure prompts, choose the prompts you wish to use in your LLM connection and save the configuration. When running an evaluation using an LLM connection, the associated prompts will automatically be attributed to each test run, allowing you to trace evaluation results to the prompts used.

The alias and version of the configured prompts will be sent to your LLM endpoint as part of the request data.

Configure Prompts

Securing Your Endpoint

If you’re concerned about the authentication of your endpoint, you can set custom headers in the LLM connection configuration. These headers will be sent with every request from Confident AI to your endpoint, which you can then use for your own authentication and authorization purposes (e.g., validating an API key).