MR Eval Gate

The MR Eval Gate runs your LLM app over a pinned dataset on every merge request, scores the outputs with a metric collection, and posts a GitLab commit status (plus a note on the merge request) that passes, fails, or is neutral depending on whether your metric scores regressed against your target branch (within your configured tolerance).

It needs two things in your project:

1. .gitlab-ci.yml — includes Confident's published CI/CD component, which sets up your app (Python + dependencies) and runs the gate on every merge request.
2. confident_eval.py — a run(input) function that calls your app and returns its output as a string. Confident calls this once per golden in your dataset.

Configure the gate

1. Connect GitLab and configure

   Go to Settings → Integrations → MR Eval Gate, connect GitLab, then choose the project, dataset, metric collection, and regression tolerance (the maximum average score drop per metric before the gate fails). Click Save.

2. Open the setup merge request

   Click Open setup merge request. Saving only stores your configuration; this step is what wires the two files into your project and adds the CONFIDENT_API_KEY CI/CD variable for you.

Setup methods

Set it up yourself

If you'd rather Confident never open a merge request for you, add everything by hand. This is the most locked-down option; you also add the API key variable yourself.

1. Include the CI/CD component

   Add Confident's component to your .gitlab-ci.yml. Set image and install_command to match your app's runtime.

   include:
     - component: gitlab.com/confident-ai/eval-gate/eval-gate@v1
       inputs:
         base_url: "<your-region-api-base-url>"
         dataset_alias: "<your-dataset-alias>"
         dataset_version: "latest"
         image: "python:3.12"                                # match your app's runtime
         install_command: "pip install -r requirements.txt"  # match your project (poetry/uv/etc.)

2. Add the eval callback

   Create confident_eval.py at the repository root. run(input) receives one dataset input, calls your app, and returns its output — Confident runs it for every golden in your dataset and scores the results.

   def run(input):
       """Return your LLM app's output for a single dataset input."""
       from my_app import agent  # import your application
   
       return agent(input)  # return the output as a string

3. Add the CI/CD variables

   Create a project API key in Settings → API Keys, then add it as a CI/CD variable named CONFIDENT_API_KEY (Settings → CI/CD → Variables in GitLab). Mask it, and leave Protect variable unchecked — protected variables are invisible to the merge-request pipelines where the gate runs. Add any runtime secrets your app needs (for example OPENAI_API_KEY) the same way.

Once the component include and confident_eval.py are on your default branch, every future merge request runs the gate and posts the Confident MR Eval Gate commit status (and a note with the score comparison) against your target branch.

Troubleshooting

Most misconfigurations fail loudly — the runner reports the reason on the Confident MR Eval Gate commit status and in the pipeline job logs. A few fail silently; those are called out below.

Last updated on July 28, 2026