Self-Hosting Changelogs

v2.5.0

Release v2.5.0 adds public dashboard sharing, model-parameter conflict handling for Anthropic and Portkey models, and several operator-facing model configuration updates. It also includes simulation capability propagation, clickable resource rows across the UI, and a few deployment and integration fixes.

Highlights

• Public dashboards can now be shared, served read-only, and queried through public resources.
• Model settings now support topK and improved handling of topP and conflicting provider parameters.
• Simulation model and capabilities are now propagated into test cases and related workflows.
• Clickable shared resource rows were added across multiple settings and observatory tables.

New Features

• Public Dashboards — Added dashboard sharing routes, public resources, and read-only rendering for publicly shared dashboards.
• Model Parameter Conflicts — Added support for deriving and exposing conflicting model parameter groups from the catalog and applying them during eval argument resolution.
• TopK Model Setting — Added a topK model setting and wired it through backend persistence and frontend submission.
• Simulation Model Propagation — Added simulation model fields and related UI support for showing the simulation model on test cases.

Improvements

• Shared Resource Rows — Added a shared ResourceRow component and reused it across multiple observatory and settings lists for consistent row interactions.
• Inline Dashboard Editing — Allowed editing a dashboard title and description inline from the dashboard page.
• Conflict-Aware Model Settings — Kept conflicting model parameters mutually exclusive in the frontend and normalized empty settings before diffing.

Fixes

• Prompt Commit TopP Handling — Stopped forcing topP when a prompt commit leaves it unset, and treated unset topP as disabled in the frontend.
• Public Dashboard Query Scoping — Fixed public dashboard queries to scope to the saved widget and requested dashboard, including correct filter handling.
• Portkey Integration — Fixed parameter conflict resolution for Portkey by resolving conflicts against the resolved gateway model.
• Saved Model Settings Cleanup — Dropped saved parameters that the selected model does not support before sending model setting updates.

Upgrade Notes

Apply the updated model catalog generation so Anthropic family-keyed settings and conflict groups are available, and review any integrations or scripts that depend on topP being defaulted. If you use public dashboards or Portkey, verify the new public resource routes and conflict resolution behavior after upgrade.

v2.4.2

Release v2.4.2 adds audit log export support, expands AI connection MCP server handling, and improves metric collection visibility in observability drawers. It also includes several fixes for export behavior, AI connection payload handling, and evaluation data propagation.

Highlights

• Audit logs can now be exported through the public API and UI with CSV output, signed downloads, and scope-aware limits.
• AI connections now support attached MCP servers, payload templates, and shared context through evals and related run paths.
• Metric collection names are now recorded and surfaced across traces, spans, threads, and observability drawers.
• Hyperparameter references in AI connection payloads are now resolved through shared editor and run helpers.

New Features

• Audit Log Exports — Added audit log export types, CSV serialization, public API routes, orchestration helpers, and recent export listings with signed downloads.
• MCP Servers for AI Connections — Added support for attaching MCP servers to AI connections, carrying their context into evals and other run types, and editing payload templates in the Resources tab.
• Metric Collection Tracking — Added support for recording every metric collection associated with traces, spans, and threads and linking them from observability drawers.
• Hyperparameter Token Resolution — Added support for resolving hyperparameter.key tokens in JSON payloads and wiring saved hyperparameter keys into the payload editor.

Improvements

• Audit Export Scope Handling — Export status records, download binding, and export run jobs are now scoped by project or organization where applicable.
• Export UX — Export dialogs now estimate audit log export size before starting a run and show recent exports with signed downloads.
• AI Connection Payload Editing — MCP server payloads and hyperparameter overrides are now handled through shared helpers and a combined Resources tab.
• Observatory Drawers — Metric collection drawers can now be opened from labels and show linked metric collection names.

Fixes

• Audit Log Export Reliability — Audit log queries are now pruned by organization and exports fall back to the payload bucket on self-hosted object storage.
• Export Run Behavior — The UI now rejects a second audit log export while one is running and expires failed exports after 10 minutes while polling active runs.
• MCP Server Payload Handling — AI connection payloads now preserve MCP servers when duplicating a connection and include them correctly in previews and warnings.
• Metric Collection Merging — Buffered trace and span payload merges now keep every metric collection and drawers skip empty metric collection names.

Upgrade Notes

Database migrations are included for audit log export enum support, MCP server relation and payload fields on AI connections, and metric collection name storage changes.

v2.4.1

This release focuses on dataset CSV parsing, validation behavior, and goldens table rendering, with one GTM dashboard enhancement. It also adds an evaluation safety check to ensure the dataset version being evaluated is validated.

Highlights

• Improved dataset CSV parsing and mapping behavior.
• Added validation of the dataset version being evaluated.
• Updated the GTM dashboard with lifecycle status and era alerts.

New Features

• GTM Lifecycle Status Card — Added a lifecycle status card, era alerts, and an active-only default to the GTM dashboard.

Improvements

• Shared JSON Parser — Extracted a shared loose JSON parser for dataset and CSV parsing paths.
• Goldens Table Context Lists — The goldens table now renders context lists.
• CSV Mapping Robustness — CSV column mapping now accepts single-quoted lists and preserves mapped columns with bad cells.

Fixes

• CSV Parsing — Fixed CSV parsing issues affecting dataset imports.
• Per-Field Validation Counts — Validation warnings now count goldens per missing field.
• Dataset Version Validation — The evaluator now validates the dataset version being evaluated.
• Optional Cell Parsing — Goldens with optional cells that fail to parse are now kept instead of being dropped.

v2.4.0

v2.4.0 centralizes model setup into consolidated /models routes, adds stricter validation and policy enforcement for credential and model updates, and expands Bedrock/Mantle support in both backend and frontend. It also includes Vertex AI routing and pricing fixes, along with audit-log and model cache updates.

Highlights

• Consolidated project and organization model routes under /models.
• Added validation for credential and model updates, including rejection of redacted placeholders and empty modelConfig objects.
• Added Bedrock Mantle support with API selection, API key auth, and optional base override.
• Improved Vertex AI routing, publisher handling, and judge pricing from the model catalog.

New Features

• Consolidated Models Routes — Added consolidated project and organization model routes under /models for reads and typed writes.
• Public Credential Updates — Added public routes to set organization and project model credentials, including project inheritance handling.
• Organization Model Configuration — Added public routes to set organization platform and model credentials.
• Project Model Configuration — Added public routes to set project evaluation models, platform model overrides, and project model credentials.
• Bedrock Mantle Support — Added Bedrock Mantle support with API selection, API key authentication, and optional API base override in backend and frontend flows.

Improvements

• Model Update Validation — Added request validators and helpers for model credential, evaluation model, and platform model updates.
• Provider Policy Enforcement — Enforced model provider policy on public model routes and public credential updates.
• Credential Cache Invalidation — Invalidated credential caches on public organization and project updates.
• Model Catalog Integration — Normalized Bedrock configs, applied live catalog metadata, and priced Vertex judges from the model catalog.
• Audit Log Scope — Kept model type information in consolidated model actions and let routes declare their own action scope.

Fixes

• Vertex Routing — Fixed Vertex model publisher and endpoint resolution, including Anthropic routing and non-Google judge handling.
• Bedrock Mantle Ping — Adjusted Mantle ping behavior to retry the alternate base instead of failing immediately.
• Simulation Write Persistence — Persisted the input token cap on simulation model writes.
• Redteam Error Propagation — Fixed redteam error propagation.
• Frontend Model Dispatch — Updated frontend model calls to use the consolidated /models route and honor forceRefresh when dispatching fetched models.

Upgrade Notes

Review any automation that calls model routes, because per-type routes were removed, the model type parameter was renamed to type, and public routes were renamed to /models. If you use Bedrock Mantle, configure the new API selection, API key, and optional API base settings before upgrade, and verify that your credentials do not rely on redacted placeholders, empty modelConfig objects, or unsupported assume-role Bedrock configs.

v2.3.2

This release adds support for matching streaming events by event name and payload type, with corresponding backend, frontend, and database updates. It also includes lifecycle, GTM, onboarding, and logging fixes, along with several internal content and layout updates.

Highlights

• Streaming event matching now supports event name and payload type/type path scoping.
• AI connection update APIs and stores now carry the new payload type/type path fields.
• Lifecycle and GTM flows received multiple fixes and sequence updates.
• Expired token failures now include redacted token logging.

New Features

• Streaming Event Matching — Streaming frames can now be matched using event name and payload type/type path across evals, ping extraction, and verify extraction.
• AI Connection Payload Type Fields — AI connection data models and backend update routes now expose payload type/type path fields for streaming configuration.
• Frontend Output Parsing Inputs — The frontend now tracks AI connection payload type/type path fields and exposes an input for output parsing event matching.
• Redacted Token Logging — Expired token failures now log redacted token information.

Improvements

• Lifecycle Sequence Updates — The full trial sequence, dormant sequence, trial ownership, cap classes, and checklist timing were updated.
• GTM Rule and Scoring Changes — GTM logic was updated for maturity scoring, phantom-send rules, since-launch ranges, and HubSpot booking scoring.
• UI and Layout Updates — The experiments and metrics layout was cleaned up, output parsing formatting was adjusted, and arena JSON displayer styling was fixed.
• Model Catalog Sync — The model catalog was synced from models.dev.

Fixes

• Lifecycle Sending Fixes — Known bouncers and the signup-abandoned double send were suppressed, and frequency cap handling was fixed.
• Partial Failure Handling — Partial failures now remain partial instead of being escalated.
• Getting Started Fix — The getting started experience was fixed.
• Onboarding Homepage Test — An A/B test was added for the onboarding homepage for new organizations.

Upgrade Notes

Database migration changes were introduced for the streaming event matching work; operators should verify the schema includes the new type path fields before upgrading dependent services.

v2.3.1

Release v2.3.1 adds Bedrock credential and trust-details support, expands risk-assessment and heatmap functionality, and introduces on-prem MCP service support. It also includes governance policy updates, cost-management schema changes, and several operator-facing fixes for Bedrock, ClickHouse, and project creation.

Highlights

• Bedrock credentials now support trust details, IAM role fields, and STS assume-role handling.
• Risk assessment adds attack heatmaps, refusal decay graphs, and related drill-down endpoints.
• Governance policy management received base-policy support and policy/schema updates.
• On-prem deployments now include an MCP service.

New Features

• Bedrock Trust Details — Added a fetcher for Bedrock trust details and loaded those details into the credentials editor.
• Bedrock IAM Role Configuration — Added Bedrock IAM role fields to the credentials editor and defaulted Bedrock to the IAM role tab while keeping both tabs visible.
• Bedrock Template Exposure — Exposed the Bedrock CloudFormation template URL and added the customer Bedrock access template.
• Risk Assessment Heatmap — Added attack-matrix and refusal-decay views, related endpoints, and heatmap UI for vulnerability analysis.
• On-Prem MCP Service — Added an MCP service for on-prem deployments.

Improvements

• Bedrock Credential Selection — Bedrock credential validation now saves per auth type and falls back to the selected model when the Bedrock config has no model ID.
• Risk Assessment Visuals — The attack heatmap and refusal-decay graphs now support pinned rows, step lines, axis titles, and improved legends.
• Governance Base Policy — Added governance base-policy support and related policy persistence changes.
• Project Creation Guard — Project creation now correctly guards on orgId.
• Model Catalog Sync — The bundled model catalog was synchronized from models.dev.

Fixes

• Bedrock Template Publishing — The Bedrock template is now published with a bucket policy instead of an ACL.
• Bedrock STS and Validation — Bedrock trust details are now required before saving an IAM role, and STS uses the chained session duration by default.
• Risk Assessment Aggregation — Attack-matrix fail rates now exclude errored test cases and overview aggregates are rebuilt for runs that never finalized.
• ClickHouse Dropdown OOM — Dropdown-related ClickHouse memory usage was reduced to avoid OOMs.
• Annotation Page Bug — Fixed a bug on the annotation page.

Upgrade Notes

Apply the included database migrations before starting the new version. Review Bedrock configuration for trust details, IAM role auth type, and STS assume-role behavior, and update any integrations that call the changed framework or model routes.

v2.3.0

Version 2.3.0 adds new trace and project concentration analysis, a new trace version/regression monitoring workflow, and several lifecycle measurement fixes. It also renames and reorganizes the monitors/alerts UI and introduces a payload type migration.

Highlights

• New concentration analysis panels and APIs for traces and project-wide signals.
• New trace version comparison, regression, and anomaly detection workflow on the monitors page.
• Lifecycle measurement and email routing fixes for enterprise deployments.
• Database migration adds payload type columns.

New Features

• Trace Concentration Analysis — Added concentration types, metadata discovery, ranking, segment counts, and a signal concentration endpoint for trace classifiers.
• Project-Wide Concentration Analysis — Added project concentration types, scan limits, inverted label distribution scanning, and a project-wide concentration endpoint.
• Trace Version Monitoring — Added trace version registries, per-version series and findings, and version comparison UI for regressions and anomalies.
• Payload Type Migration — Added payload type columns to AI connection storage.

Improvements

• Concentration UX — Added shared bars, sparklines, grouped segment rendering, stats cards, and clearer concentration copy and labels across the monitor UI.
• Version Significance Scoring — Added chi-square and Benjamini-Hochberg significance handling, corrected confidence display, and bounded ratio scoring for version findings.
• Lifecycle Measurement — Improved lifecycle tracking with server-side arrival beacons, more accurate click timing, and stricter follow-up behavior.
• Monitors Naming — Renamed the alerts page and related copy to monitors, with the concentration section renamed to what stands out.

Fixes

• Concentration State Handling — Fixed aborted request handling, panel cleanup, empty range detection, and error states for concentration views.
• Trace Filtering — Fixed metadata value filtering and span-derived filter handling in the trace list query builder.
• Version Filtering — Fixed version dropdown search, baseline handling, and preservation of unversioned hash filters in segment counts.
• Lifecycle Email Routing — Fixed lifecycle email targeting so enterprise emails reach recipients and the checklist and dormant follow-up flows match their intended steps.

Upgrade Notes

Apply the database migration for payload type columns before starting the new release. If you have scripts, dashboards, or bookmarks that reference the old alerts naming, update them to monitors.

v2.2.0

This release updates the self-hosting release workflow and adds Helm chart version v0.2.0. No application-level runtime changes are indicated beyond deployment packaging updates.

Highlights

• Added Helm chart version v0.2.0.
• Updated the release workflow for self-hosting.

New Features

• Helm Chart v0.2.0 — A new Helm chart version v0.2.0 is available for self-hosted deployments.

Improvements

• Release Workflow Update — The release.yml workflow was updated.

Upgrade Notes

Operators using Helm should review the new chart version v0.2.0 before upgrading; no additional runtime or database migration steps are indicated in the input.

v2.1.2

This release adds self-hosted signup disablement with admin seeding, expands audit logging and attribution, and introduces governance runtime controls for metric and annotation data. It also includes ClickHouse and audit-log fixes, plus several migration and deployment-related changes that require operator attention.

Highlights

• Self-hosted signup can now be disabled and replaced with a seed-admin bootstrap flow.
• Audit logs gain better organization/resource attribution and optional stdout mirroring on on-prem deployments.
• Governance runtime controls now support metric data and annotations, including metric-level selection and persistence of extra query params.
• New database migrations add model provider policy and governance control version fields.

New Features

• Disable Sign-Up Flow — The backend and frontend now support disabling email/password signup via configuration and routing users to login when signup is disabled.
• Seed Admin Accounts — A seed-admin script was added to pre-create bootstrap admin accounts for self-hosted deployments.
• Audit Log Stdout Mirroring — On-prem deployments can now mirror audit events to stdout when the new audit log stdout setting is enabled.
• Governance Runtime Controls Expansion — Governance runtime controls now support metric data, annotations, metric-level selection, median score aggregation, and extra query params on control versions.

Improvements

• ClickHouse Span Read Optimization — The backend no longer performs wasted full-IO reads for offloaded span payloads in ClickHouse.
• Audit Log Attribution — Audit log actor, organization, project, invitation, and token acceptance attribution were tightened when request parameters are missing or overridden.
• HubSpot PQL Sync — Qualified lead flow now enters HubSpot at onboarding completion and syncs the PQL score explanation to HubSpot.
• Governance Editor Consistency — The governance UI now exposes metric data and annotations in the runtime control editor and uses consistent data-model labeling.

Fixes

• Atomic Admin Seeding — Admin seeding is now atomic to avoid partial user creation.
• Organization Fallback In Audit Logs — Routes without an organizationId now fall back to the actor's organization for audit logging.
• Cost Display Corrections — Cost display was fixed for multi-turn test runs with traces.
• Upgrade Callout UI — The frontend callout component was upgraded.

Upgrade Notes

Apply the new database migrations before starting the upgraded release. If you use self-hosted signup disablement, configure DISABLE_SIGN_UP and NEXT_PUBLIC_DISABLE_SIGN_UP or the Helm disableSignUp value; if you use audit stdout mirroring, configure AUDIT_LOG_STDOUT_ENABLED or the Helm auditLogStdout value.

v2.1.1

This release adds API key expiry and rotation support, new golden dataset/public endpoints, and several observability and discovery updates. It also includes security logging hardening and database migrations that operators should plan for.

Highlights

• API keys now support expiry, rotation, grace periods, and status display.
• Golden endpoints were split to support stable IDs and new public read/write operations.
• Async test runs, discovery setup, and observatory onboarding received user-facing updates.
• This release includes schema migrations for API keys and trace/full-IO related changes.

New Features

• API Key Expiry And Rotation — API keys can now carry an expiration date, be rotated in place, and surface expiry and status information in settings tables and public v1 responses.
• Golden Endpoint Split — Golden routes now use a stable dataset identifier and expose new public update, delete, read, and post endpoints.
• HubSpot CRM Sync — Signups, organizations, plans, memberships, onboarding answers, seat counts, lifecycle stages, company domains, and Stripe customer IDs are now synced to HubSpot.
• Discovery Setup Guides — Setup guides were added for tracing and test runs as part of the discovery revamp.
• Default Report Templates — Default report templates were added.

Improvements

• Async Test Run Events — SSE events for async test runs now show waiting test cases, and test case warnings and statuses were updated.
• Observability Onboarding — Observatory onboarding and feature gating were improved.
• Governance Policy UI — The add button was restored in governance policies.
• Marketing Materials Page — A Marketing Materials page was added to the GTM dashboard.

Fixes

• Security Logging — Credential bundles, raw axios errors, SAQ job kwargs, and model configuration details are no longer logged in sensitive failure paths.
• Api Key Cache Invalidation — API key caches are now invalidated on public, client, and organization key updates, deletes, and rotations.
• Trace Full IO — Trace and spans public endpoints now support fullIO handling.
• Platform Model Ping — The platform model ping path was fixed.
• Classification Graphs — Classification graphs were fixed, including thread classification graphs and chart updates.
• Online Metrics Parsing — Online metrics now correctly parse tool call types.
• Educational Email Handling — Educational emails are now allowed across signup, login, CLI auth, and invites.
• Pydantic Schema — The pydantic schema was updated for a new content type.
• Security Provider Keys — Provider key handling was fixed.

Upgrade Notes

Apply the new API key migrations before deploying this release, and verify any automation that manages API keys, golden endpoints, or trace/span public endpoints against the updated contracts.

v2.1.0

Release v2.1.0 expands organization-level billing and cost-insights, adds invitation token flows for onboarding and public invite routes, and introduces AWS Marketplace license-manager integration. It also includes test-run cost and latency reporting, metric evaluation model support, and several deployment and packaging updates.

Highlights

• Organization billing now includes cost-insights, cost breakdown, usage breakdown, and cost-by-project views.
• Invitation links now use tokenized public routes and support token preview and acceptance flows.
• Test runs now expose cost and latency graphs and per-run trace cost aggregation.
• Helm and backend now support AWS Marketplace license-manager deployment and licensing configuration.
• Standardized evaluation pings, model validation
• Added flaky metric and nullable threshold support

New Features

• Organization Cost Insights — Added organization-level cost-insights, cost breakdown, usage breakdown, billed-cost, and cost-by-project endpoints and UI pages.
• Tokenized Invitations — Added invitation tokens, public token preview and accept endpoints, and tokenized invite emails for org and project invitations.
• Metric Evaluation Model — Added metric evaluation model support, including per-metric model resolution and frontend updates for metric-specific evals.
• Test-Run Cost And Latency — Added test-run cost and latency routes, graphs, and per-run trace cost aggregation in test-run insights.
• AWS Marketplace Licensing — Added AWS Marketplace license-manager integration, licensing mode configuration, and Helm support for marketplace deployment.
• Standardized Evaluation Pings — Evaluation pings have been standardized.
• Model Validation Hook — A model validation hook was added.
• Flaky Metric Support — Support was added for flaky metrics and nullable thresholds.

Improvements

• Usage Helpers By Project — Updated cost-by-period and usage-count helpers to accept project IDs.
• Onboarding Invite Flow — Added a team invite onboarding step with per-member role selectors and grouped invite sending.
• Test-Run Graph Layout — Adjusted test-run performance and cost graph layout, tooltip syncing, and empty-state behavior.
• Helm Packaging And Images — Updated Helm chart image handling, packaging scripts, and single-architecture image support.

Fixes

• Cost Breakdown Filtering — Excluded unmetered AI-feature cost from organization cost breakdown and limited offline-eval cost to customer-keys only.
• Test-Case And Eval Fixes — Fixed test-case result logging and several metric evaluation model issues, including provider defaults.
• Invitation UI Fixes — Adjusted invite notice, invite email, and onboarding button layout and visibility behaviors.
• OpenTelemetry Build — Fixed the OpenTelemetry build and added logging support.

Upgrade Notes

Apply the database migration that adds the invitation token column before enabling the new invite flow. If using AWS Marketplace licensing, configure LICENSE_MODE, the AWS Marketplace product settings, and the injected license secret/service account in Helm values. Update any deployments that reference the removed sample rate fields or the old combined registry/tag image setting.

v2.0.20

Release v2.0.20 adds classification polarity, export streaming/compression, async AI responses, and cost/billing updates. It also includes operator-facing migration and deployment changes for schema, Helm, and multi-architecture builds.

Highlights

• Classification labels now support polarity and tone across label creation, signal findings, and trends.
• Exports can now stream and upload gzipped files to supported object stores, with a higher manual export cap for streamed uploads.
• AI connections now support async response mode and the UI/API wording has been updated accordingly.
• Cost insights and billing logic were updated to use per-organization grouping and new price IDs, with backfill support.

New Features

• Classification Polarity — Classifier labels now carry a polarity value that is exposed in label, finding, and label-generation flows.
• Async AI Responses — AI connections now support async response mode, and the endpoints and UI were updated to use the new naming.
• Streaming Exports — Trace and thread exports now stream uploads to S3, MinIO, GCS, and Azure Blob Storage, with gzipped export support.
• Cost Insights — Cost insights now include updated cost tiles, tabs, and funding-source breakdowns in the frontend.

Improvements

• Multi-Architecture Builds — The build pipeline now produces multi-architecture images.
• Export Reliability — Export workers now enforce a byte ceiling, clean up failed uploads, and destroy gzip streams on upload failure.
• Classification Storage — Signals, traces, and threads now read classifier data from the classifications table instead of the labels map.
• Annotation Filtering — Annotations now support date filtering and a created-versus-updated toggle.

Fixes

• Report PDF Layout — Report generation now respects start-on-new-page settings and updates risk assessment title and table-of-contents styling.
• Trial Free-Plan Guard — Expired trials can no longer access the Free plan server-side and the Free option is hidden in the UI.
• Signal and Thread UI — Signal findings, label rows, and thread chat views now render polarity, tone, and last-message information correctly.
• Pricing Instrumentation — Pricing events now carry measurable plan and selection data, and blank pricing events are filtered out.

Upgrade Notes

Apply the database migrations for classification labels, signal polarity, and code-scan-run before starting the new release. Review updated Helm/deployment artifacts, new price IDs, and the per-organization billing cutover behavior before upgrading.

v2.0.19

Release v2.0.19 adds span sampling support, GitHub App support for DeepEval, and a new classifications data path with ClickHouse storage, buffered inserts, dual-write, and backfill migration work. It also includes database migration work for AI connections and usage features.

Highlights

• Added span sample rate support.
• Added DeepEval support for GitHub Apps.
• Introduced classifications storage, buffering, dual-write, and backfill changes.
• Included new database migrations for AI connection, usage, and classifications features.

New Features

• Span Sample Rate — Added support for configuring a span sample rate.
• DeepEval GitHub Apps — Added DeepEval support for GitHub Apps.
• Classifications Table — Added a classifications ClickHouse table with schema, types, and migrations.
• AI Connection and Usage Migrations — Added migration 34 for AI connection and label generation usage features.

Improvements

• Classifications Buffering — Added a buffered insert service for classifications.
• Classifications Dual Write — Enabled dual-write of classifications alongside the existing trace/thread labels map.
• Classifications Backfill — Added a backfill script for classifications data.

Upgrade Notes

This release includes new database migrations, including migration 34 and migration 35 for classifications; operators should run the full upgrade and data-migration steps before enabling the new classifications path.

v2.0.18

Release v2.0.18 adds new Helm chart packaging, MCP OAuth/auth configuration support, and expanded widget-query endpoints for project and organization scopes. It also includes entitlements route gating updates, risk assessment and sampling enhancements, and several operator-facing fixes.

Highlights

• New Helm chart scaffold and deployment templates are included for self-hosted installs.
• MCP server authentication now supports OAuth client-credentials and headers-based auth with schema and UI updates.
• Widget queries were expanded to dedicated project and organization endpoints, with backend routing changes.
• New sampling, email, and risk-assessment updates were added across workflows and evaluations.

New Features

• Helm Chart — A new Helm chart scaffold was added with deployment templates, external-secrets integration, Redis and ClickHouse resources, ingress, and backup jobs.
• MCP OAuth Auth — MCP server auth now supports OAuth client-credentials and headers-based configurations, including backend persistence, redaction, and editor support.
• Widget Query Endpoints — New widget-query APIs were added for project and organization scopes, with backend support for building and serving aggregate queries.
• Sampling Controls — Thread and trace sample-rate support was added, along with workflow sampling inputs and queue sampling for evals.
• Email Reporting — Email configuration and report-email delivery logic were added, including support for email report bodies.
• Code Execution Support — A GCP Cloud Functions executor was added to the backend and evals.

Improvements

• Entitlements Gating — Evals and observability client routes were consolidated under single-source path-based feature gating.
• Widget Query Routing — Dashboard and aggregate fetchers were routed through the widget endpoint, with memoized caching added for fetchers.
• Risk Assessment — Risk assessment now starts via ID and includes fallback error handling.
• MCP Connect Feedback — MCP connect now returns a failure reason that is surfaced in the frontend.

Fixes

• Widget Query Stability — Widget query failures are now logged per query, and abort behavior in NONE cache mode was corrected.
• Widget Query Validation — Org widget queries now honor per-query projectId filters, enforce batch and nested time-range bounds, and reject unsupported data models.
• Evaluation Reload — The evaluate UI now reloads correctly for spans, traces, and threads.
• Preserve Annotator Buckets — Annotator buckets are preserved for removed users.

Upgrade Notes

Apply the new database migrations for widget-query and MCP auth changes before restarting application components. If you manage MCP servers or widget-query clients directly, update request payloads and endpoint paths to match the new APIs.

v2.0.17

This release adds new AI connection logging and configuration capabilities, expands dashboards and reports, and introduces several trace, test run, and DeepTeam workflow updates. It also includes database migrations and UI route renames that operators should account for during upgrade.

Highlights

• AI connection logs and log tables were added, along with new AI connection configuration options.
• Dashboards and reports gained new public endpoints, widget endpoints, and annotation graph support.
• Trace and test run workflows received require-review, export, persistence, and performance updates.
• Several migrations and route changes were introduced, including AI connection query params and /setup to /connect renaming.

New Features

• AI Connection Logs — AI connection logs and a dedicated AI connection log table were added.
• GitHub App Connection — GitHub App connection support was added, including routes and a connection page.
• Dashboard Public APIs — Dashboard public endpoints and widget get endpoints were added.
• Risk Assessment Endpoints — Public endpoints for the risk assessment framework were added.
• Thread Exports — Thread export support was added.
• Release Notes in AI Connection — AI connection now supports release notes and array output.

Improvements

• Reports And Navigation Rename — Insights was renamed to reports, and the setup route was refactored to /connect.
• AI Connection Configuration — AI connection configuration now supports save-all behavior, draft state, query params, and AI feature config.
• Dashboard and Annotation UX — Status pills, collapsible sections, clickable URLs, and improved annotation filter presentation were added.
• Trace And Test Run Performance — Trace and test run processing gained persistence in Redis, merge helper reuse, and thread aggregate speed improvements.
• Token Handling — The platform model page now exposes input and output token settings, and signal handling now supports max_input_tokens without trace truncation.
• DeepTeam Workflow Updates — DeepTeam gained bot comment methods and workflow fixes for empty repositories.

Fixes

• Annotator And User Redirects — Annotator selection and user page redirects were fixed to preserve filters and resolve names via the global user table.
• Histogram And Statistics UI — Histogram bars and statistics tab pill rendering were fixed.
• Multi-Gen Test Cases — Multi-gen test cases no longer disappear and test case group loading was improved.
• DeepTeam Validation — DeepTeam zod validation was fixed.
• Report Template Bugs — Report template bugs were fixed.
• OpenAI Fetch Handling — OpenAI now uses the native Node.js fetch implementation instead of node-fetch.
• API Keys Reveal Query — The API keys endpoint now supports a reveal query parameter.

Upgrade Notes

Apply the new database migrations for GitHub App connections, AI connection logs, AI connection query params, and trace requireReview support before rolling out the new build. If you expose the app externally, update routes and documentation to account for the /setup to /connect rename and the insights to reports rename.

v2.0.16

Migrates dashboards to the unified Widget data model and adds a governance controls page, backed by a supporting database migration.

New Features

• Governance Controls Page — New page to configure and review governance controls across projects.
• Widget Data Model — Dashboards migrated to a unified Widget model, replacing the legacy Graph model.

Improvements

• Governance Runs Consolidation — Removed standalone governance runs in favor of the controls model.

Fixes

• Risk Assessment Pages — Fixed rendering on risk-assessment pages.

Upgrade Notes

Run the Graph-to-Widget migration before starting this version; it backfills existing dashboards onto the Widget model.

v2.0.15

Adds paused-service notifications and governance policy improvements, byte-bounded ingestion buffers, risk-assessment cost tracking, and several fixes.

New Features

• Paused-Service Notifications — In-app notices when a service such as signals or online evaluations is paused, with guidance on how to proceed.
• Governance Policy Improvements — Expanded governance policy engine and controls.
• Risk Assessment Cost Tracking — Track LLM cost attributed to risk assessments.

Improvements

• Byte-Bounded Ingestion Buffers — Ingestion buffers are now bounded by byte size for more predictable memory usage.
• Standardized AI Streaming — Unified server-sent-event streaming across AI features.

Fixes

• Report Rendering — Fixed report rendering.
• UUID Parsing — Fixed UUID parsing.
• AI Connections — Fixed an AI connection issue.

v2.0.14

Launch-week release. Adds AI-generated metric criteria and rubrics, per-evaluation model overrides, classifier filters and workflow chaining, executive insight and risk-assessment reports, dashboard templates, AI-feature usage and cost tracking, and the ability to pause online metric evaluations.

Highlights

• AI-generated metric criteria and rubrics
• Per-evaluation model overrides
• Executive Insight and Risk Assessment reports
• AI-feature usage and cost tracking

New Features

• AI Criteria & Rubric Generation — Metrics can generate their own criteria and scoring rubrics using LLMs.
• Per-Evaluation Model Overrides — Select the evaluation model per metric, with platform- and feature-specific defaults.
• Classifier Filters & Workflow Chaining — Classifiers support saved filter configurations and downstream workflow chaining.
• Executive Insight Reports — Generate executive-level insight reports, including support for custom (BYOK) models.
• Risk Assessment Reports — Templated risk-assessment reporting.
• Dashboard Templates — Prebuilt dashboard templates.
• AI-Feature Usage & Cost Tracking — Track LLM usage and cost per AI feature, including signals.
• Pause Online Metric Evaluations — Operators can pause online metric evaluations per project.
• Red Teaming Controls — Governance controls for red teaming.
• Offloaded Span I/O — Spans can include offloaded input/output payloads.
• Organization Invitations — Improved organization invitation flow, including select-all.

Improvements

• AND/OR Filter Groups — Filters support grouped AND/OR logic.
• Faster Graphs — Reduced graph and metric-chart load times on high-volume projects.
• Report Standardization — Standardized the reporting pipeline and removed legacy report logic.
• Evaluation Model Defaults — Updated default evaluation model to gpt-5-nano for lower cost and latency.
• Model Call Logging — Added logging for model and AI-connection requests.

Fixes

• Redis Ingestion Buffers — Fixed Redis ingestion buffer handling.
• Experiments — Fixed experiment alignment and a rendering issue.
• Risk Assessment — Fixed a risk-assessment bug.
• Classifiers — Fixed classifier issues.
• Alerts Styling — Fixed alerts styling.
• Logging Noise — Reduced excessive log output.

Upgrade Notes

This release includes a database migration for AI-feature usage tracking; run pending migrations before starting the new version. The default evaluation model changed to gpt-5-nano — review this if you rely on a pinned evaluation model.

v2.0.13

Introduces flexible scheduling with onset, recurrence, and termination controls, extends triage to spans, threads, and test runs, adds official test runs and annotation queues, and ships several scheduling and alert fixes.

New Features

• Flexible Scheduling — Custom schedule settings with configurable onset, recurrence, and termination for tasks and alerts.
• Triage for Spans, Threads & Test Runs — Ticket creation and triage now extend beyond traces to spans, threads, and test runs.
• Official Test Runs — Mark canonical test runs as official so scratch runs no longer pollute evaluation history.
• Annotation Queues — Annotation forms now support review queues.

Fixes

• Bedrock ARN Parsing — Fixed AWS Bedrock ARN parsing.
• Alerts Validation — Fixed a validation error when configuring alerts.
• Scheduling UI — Fixed toast and hydration issues in the scheduling UI.
• Project Settings UI — Fixed rendering of the project settings page.

Upgrade Notes

Run the schedule onset/recurrence/termination migration and the annotation-form response migration before starting this version.

v2.0.12

Adds configurable annotation forms and expands scheduled-task settings, each backed by a database migration.

New Features

• Annotation Forms — Capture structured human feedback through configurable annotation forms and criteria.
• Scheduled Task Settings — Expanded configuration options for scheduled tasks.

Upgrade Notes

Run the pending database migrations for expanded schedule settings and annotation-form criteria before starting this version.

v2.0.11

Finalizes the scheduling-settings migration by dropping legacy columns, adds official risk assessments, and hardens streaming JSON parsing.

New Features

• Official Risk Assessments — Mark risk assessments as official to separate canonical runs from scratch runs.

Improvements

• Robust JSON Parsing — Extract JSON even when wrapped in markdown or model reasoning, rather than failing on strict parsing.

Fixes

• Streaming JSON — Fixed streaming JSON parse errors.

Upgrade Notes

Drops legacy scheduling columns; run the drop-legacy-scheduling-columns migration after the v2.0.10 scheduling migration.

v2.0.10

Rebuilds the alerts page with full history, adds pannable graphs, and lays the groundwork for flexible scheduling.

New Features

• New Alerts Page — Rebuilt alerts page where each alert retains its full history.
• Pannable Graphs — Graphs can now be panned.

Improvements

• Metadata Counts — Fixed count deduplication and added metadata counts.

Upgrade Notes

Run the scheduling-settings migrations before starting this version.

v2.0.9

Adds pre-deployment governance controls, an organization-level API, workflow graphs, test-run and trace filters, and AWS inference support, with data-model cleanup migrations.

Highlights

• Pre-deployment governance controls
• Organization-level API endpoints
• Test-run and trace filtering

New Features

• Pre-Deployment Controls — Governance controls that run before deployment.
• Organization API Endpoints — Organization-level endpoints for programmatic provisioning and management.
• Workflow Graphs — Graph view for workflows.
• AWS Inference Support — Added AWS inference as a model provider option.
• Test Run Filters — Filter test runs.
• Trace Tool Filter — Filter traces by tool.
• Project & Organization Settings — Reorganized project and organization settings.

Improvements

• Multi-Root Ingestion Pipeline — Support for multi-root trace ingestion.
• Model Costs — Updated model cost data.

Fixes

• Risk Assessment — Fixed filter pagination and an assessment bug.
• Annotation Queue — Fixed annotation-queue handling.
• Multi-Turn Simulation — Fixed multi-turn simulation display.
• UI — Fixed dialog width, button timeouts, and color palette.
• Config Types — Fixed configuration type handling.

Upgrade Notes

This release includes several migrations (v2.0.8 data migration, startAt, usage token count) and drops stale model tables. Run pending migrations and back up your database before upgrading.

v2.0.8

Introduces AI Governance, vulnerability trace scanning and threat detection, GitHub and Linear ticketing integrations, alert history and logs, dataset-version API support, and a unified models data model.

Highlights

• AI Governance (initial release)
• Vulnerability trace scanning and threat detection
• GitHub / Linear ticketing integrations
• Alert history and logs

New Features

• AI Governance — Initial governance capabilities for policies and controls.
• Vulnerability Trace Scanning — Scan ingested traces for security vulnerabilities.
• Trace Threat Detection — Detect threats in ingested traces.
• Traces on Red Teaming Test Cases — Red-teaming test cases now include full traces.
• GitHub & Linear Integrations — Push problem traces into GitHub or Linear as tickets, with support for multiple integrations.
• Alert History & Logs — Full history and logs for every alert that fires.
• Dataset Version API — Manage dataset versions via the API.
• Bulk Export — Bulk export of platform data.
• New Data Sources — Additional knowledge-base data sources.
• Custom Headers for AI Connections — Support custom headers on AI connection requests.

Improvements

• Unified Models Data Model — Migrated to a consolidated models table.
• API Keys — Refactored API-key handling.
• Experiments — Enhancements and metric-map fixes.
• Graph Layout — Refactored graph layout.

Fixes

• Thread Metadata — Fixed thread-metadata filtering and merging.
• Thread Styling — Fixed thread styling.

Upgrade Notes

Run the models migration and the numGeneration/generationOrder migration before starting this version. The Australia (AU) server is deprecated in this release.

v2.0.7

Adds live risk-assessment updates, annotation-queue assignment and notifications, invitation roles, dashboard and report exports, and thread metadata throughout the stack, alongside a large set of ingestion and reliability improvements.

Highlights

• Risk assessment live updates
• Annotation queue assignment and notifications
• Dashboard and report PDF / image export
• Major ingestion reliability work

New Features

• Risk Assessment Live Updates — Attack methods and vulnerabilities surface live during a risk assessment.
• Queue Assignment & Notifications — Annotation queues route work to specific teammates and notify the assignee.
• Invitation Roles — Assign a role when inviting users to an organization.
• Dashboard & Report Export — Export dashboard widgets and reports to PDF or image.
• Thread Metadata Everywhere — Thread metadata flows through ingestion, data tables, filters, and dashboards.
• Turn Limits for Threads — Configurable limits on conversation turns.
• Prompt on LLM Spans — LLM spans now display the prompt that was used.

Improvements

• Ingestion Reliability — New ingestion workers, head-of-line-blocking fixes, and I/O chunking for large payloads.
• Postgres Connection Pooling — Resolved connection-pool exhaustion under load.
• Classification Queue — Moved the classification queue to Python for reliability.
• Event Tracking — Refactored event tracking.
• Cost Insights — Revamped the cost insights page.

Fixes

• Two-Factor Authentication — Restored 2FA.
• Metric Scoring — Fixed metric-score-by-prompt and metric-data timeouts.
• Model Temperature — Fixed temperature handling for Claude models.

Upgrade Notes

Run the assignment, invitation-role, and related migrations before starting this version.

v2.0.6

Adds evaluation rules for running online evaluations from the UI, multimodal traces, model-cost tracking, and WebSocket support, with reliability fixes.

New Features

• Evaluation Rules — Configure and run online evaluations directly from the UI, without writing API calls.
• Multimodal Traces — Traces support PDFs and images in inputs and outputs.
• Model Cost Tracking — Track model costs across evaluations and traces.
• WebSocket Support — Real-time updates delivered over WebSocket connections.
• Revamped Evaluation Creator — Rebuilt the evaluation creation flow.

Improvements

• Online Metric Observability — Added observability for online-metric evaluations.
• Arena Experiment Support — Improved Arena support within experiments.
• Connection Pooling — Dedicated worker connection pooling for stability.

Fixes

• Evaluation Rules — Fixed duplicate eval-rule metrics and assorted eval-rule bugs.
• Observability — Fixed a trace ingestion issue.
• Framework Creation — Fixed framework creation.

Upgrade Notes

Run the eval-rules and eval-rule-override migrations before starting this version.