PR Eval Gate

The PR Eval Gate runs your LLM app over a pinned dataset on every pull request, scores the outputs with a metric collection, and posts a GitHub check-run that passes, fails, or is neutral depending on whether your metric scores regressed against your base branch (within your configured tolerance).

It needs two things in your repository:

1. .github/workflows/confident-eval-gate.yml — a workflow that sets up your app (Python + dependencies) and runs Confident’s published runner Action.
2. confident_eval.py — a run(input) function that calls your app and returns its output as a string. Confident calls this once per golden in your dataset.

Configure the gate

Setup methods

Set it up yourself

If you’d rather Confident never open a PR for you, add both files by hand. This is the most locked-down option; you also add the API key secret yourself.

1
name: Confident PR Eval Gate
2
3
on:
4
  pull_request:
5
  push:
6
    branches: ["<your-default-branch>"]
7
8
permissions:
9
  contents: read
10
11
jobs:
12
  eval-gate:
13
    runs-on: ubuntu-latest
14
    env:
15
      # Any runtime secrets your app needs to run, referencing repo secrets, e.g.:
16
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
17
    steps:
18
      - uses: actions/checkout@v4
19
      - uses: actions/setup-python@v5
20
        with:
21
          python-version: "3.12" # match the version your app targets
22
      - name: Install dependencies
23
        run: pip install -r requirements.txt # match your project (poetry/uv/etc.)
24
      - name: Confident PR Eval Gate
25
        uses: confident-ai/deepeval-tracer/actions/eval-gate@v1
26
        with:
27
          base_url: "<your-region-api-base-url>"
28
          dataset_alias: "<your-dataset-alias>"
29
          dataset_version: "latest"
30
          confident_api_key: ${{ secrets.CONFIDENT_API_KEY }}

1
def run(input):
2
    """Return your LLM app's output for a single dataset input."""
3
    from my_app import agent  # import your application
4
5
    return agent(input)  # return the output as a string

Once the workflow and confident_eval.py are on your default branch, every future pull request runs the gate and posts the Confident PR Eval Gate check-run with the score comparison against your base branch.

Troubleshooting

Most misconfigurations fail loudly — the runner reports the reason on the Confident PR Eval Gate check-run and in the workflow’s Actions logs. A few fail silently; those are called out below.