PR Eval Gate

The PR Eval Gate runs your LLM app over a pinned dataset on every pull request, scores the outputs with a metric collection, and posts a GitHub check-run that passes, fails, or is neutral depending on whether your metric scores regressed against your base branch (within your configured tolerance).

It needs two things in your repository:

1. .github/workflows/confident-eval-gate.yml — a workflow that sets up your app (Python + dependencies) and runs Confident's published runner Action.
2. confident_eval.py — a run(input) function that calls your app and returns its output as a string. Confident calls this once per golden in your dataset.

Configure the gate

1. Connect GitHub and configure

   Go to Settings → Integrations → PR Eval Gate, install the Confident GitHub App on your repository, then choose the repository, dataset, metric collection, and regression tolerance (the maximum average score drop per metric before the gate fails). Click Save.

2. Open the setup pull request

   Click Open setup pull request and pick a setup method (below). Saving only stores your configuration; this step is what wires the two files into your repo.

Setup methods

Set it up yourself

If you'd rather Confident never open a PR for you, add both files by hand. This is the most locked-down option; you also add the API key secret yourself.

1. Add the CI workflow

   Create .github/workflows/confident-eval-gate.yml. Set up your app's runtime (Python + dependencies) in earlier steps, then invoke the runner Action — keep the final Confident PR Eval Gate step's uses: ref and its four with: inputs.

   name: Confident PR Eval Gate
   
   on:
     pull_request:
     push:
       branches: ["<your-default-branch>"]
   
   permissions:
     contents: read
   
   jobs:
     eval-gate:
       runs-on: ubuntu-latest
       env:
         # Any runtime secrets your app needs to run, referencing repo secrets, e.g.:
         OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
             python-version: "3.12" # match the version your app targets
         - name: Install dependencies
           run: pip install -r requirements.txt # match your project (poetry/uv/etc.)
         - name: Confident PR Eval Gate
           uses: confident-ai/deepeval-actions/actions/eval-gate@v1
           with:
             base_url: "<your-region-api-base-url>"
             dataset_alias: "<your-dataset-alias>"
             dataset_version: "latest"
             confident_api_key: ${{ secrets.CONFIDENT_API_KEY }}

2. Add the eval callback

   Create confident_eval.py at the repository root. run(input) receives one dataset input, calls your app, and returns its output — Confident runs it for every golden in your dataset and scores the results.

   def run(input):
       """Return your LLM app's output for a single dataset input."""
       from my_app import agent  # import your application
   
       return agent(input)  # return the output as a string

3. Add the repository secrets

   Create a project API key in Settings → API Keys, then add it as a repository secret named CONFIDENT_API_KEY (Settings → Secrets and variables → Actions in GitHub). Add any runtime secrets your app needs (for example OPENAI_API_KEY) the same way.

Once the workflow and confident_eval.py are on your default branch, every future pull request runs the gate and posts the Confident PR Eval Gate check-run with the score comparison against your base branch.

Troubleshooting

Most misconfigurations fail loudly — the runner reports the reason on the Confident PR Eval Gate check-run and in the workflow's Actions logs. A few fail silently; those are called out below.

Last updated on July 27, 2026