Prompt-ly Evaluated
TGIF! Thank god it’s features, here’s what we shipped this week:
Headline first: Prompt Evals are here. Think GitHub Actions, but for prompt commits and version releases—so every prompt change can trigger the checks that keep quality high and surprises low.
Added
- Prompt Evals - The big one. Run evals on prompt commits and version releases automatically—CI for prompts, not vibes-based QA.
- Support Ticket Submission Page - Need help? There’s now a dedicated place to ask for it.
- Trace Classification - Sort your traces into categories. Less chaos, more class.
- Dataset Threads + Scenario Generation - Datasets now support threads, with scenario generation to spin up richer test cases.
- Portkey Support in Arena and Experiments - Portkey now works in Arena and Experiments. The key to connected workflows.
- SSE + HTTP Streaming for AI Connections - Stream responses over SSE or HTTP. Go with the flow.
- Azure Key Vault Integration - Store secrets in Azure Key Vault. Your keys, under lock and cloud.
- Org Settings Pages - New pages for roles, permissions, and API keys. Access control, finally under control.
Changed
- Evaluate Buttons on Traces and Spans - Trigger evals directly from where issues appear, so troubleshooting is fewer clicks and more signal.
