Evaluation Studio

Tests you own

Agents Studio ships with template eval packs. Evaluation Studio, planned for W3, lets a tenant build its own. In Year 1, the eval harness and Eval Coverage Score inside Agents Studio do this job.

Suites and scorers

Eval suites exist per skill and agent, with tenant-authored golden tasks and severity tags. Golden tasks can be imported from real tickets, deals and conversations with redaction. Scorers include exact match, schema validity, rubric and LLM-as-judge, with the judge calibrated against human labels. A judge calibration dashboard tracks agreement with human labels and drift, and scorer disagreements go to a human review queue.

Regression and gates

Eval runs on candidate versions detect regressions against the current version. Promotion gates can be configured per autonomy level, but only stricter than platform thresholds, never looser. Held-out sets and contamination control keep evals separate from training data. A cost and latency Pareto view compares model routes that pass the same evals. CI integration runs evals on pull requests for tenant packs through the Developer Platform.

Red-teaming

Red-team suites test injection through email, web and documents, data exfiltration, jailbreaks, regulated claims and disclosure behaviour. Results break down by language: English, Hindi, Tamil, code-mixed and noisy transcripts.

Always on

Production QA monitors sample live sessions and score them continuously, so quality problems surface between releases, feeding themes and alerts in Agents Studio.

How it fits

Evaluation Studio supplies the suites that Agents Studio uses for promotion, that the Prompt & Skill Library uses to compare versions, and that the Performance Arena uses as evidence for certification. Synthetic data from Synthetic Data Studio and scenarios from the Training Gym can become golden tasks, so tenants can test sensitive workflows without exposing real customers.