AI Bench
AI Bench sends a set of sample prompts through your own gateway and model, with your current guardrails active, and scores what came back against the expected outcome. It gives you a measured pass rate for a given model and guardrail configuration, and a per-sample record of what passed and what did not.
AI Bench is its own console section, visible when your edition includes the
benchmarks feature. Viewing runs and starting them are separate
permissions.
How a run works
Cortega sends the prompts itself, through a gateway and model you select, using an internal shared credential. Production traffic is not involved.
The run does not create, change, or delete guardrails. It records the guardrails that are active at the time of the run and scores against that snapshot.
Every sample is a real governed call, so a run also surfaces connectivity and routing problems: a wrong provider key or an unreachable model shows up as errored samples.
Industries and suites
Each run is scoped to an industry, which selects the sample set.
| Industry | What it tests |
|---|---|
| General safety | Cross-policy safety categories: hate, harassment, self-harm, violence, sexual content, and their more severe variants |
| Healthcare | Triage boundaries, medication safety, PHI handling, emergency escalation |
| Finance | Unqualified financial advice, fraud enablement, market manipulation, financial PII |
| Legal | Legal-advice boundaries, confidentiality, evidence handling, and whether the model's own case-law citations are real (see below) |
| Privacy / PII | Whether the model discloses synthetic PII and PHI it should redact, across identity, contact, credential, and health or financial entities |
| Security / red-team | Jailbreak templates, prompt injection, credential abuse, cyber-harm behaviors, and a benign set that measures over-refusal |
An industry can have more than one suite; the run form picks one. Each suite is a curated set of sample prompts, built and refined over time from public and research safety benchmarks.
Modes
The mode sets how far down the request and response pipeline each sample runs.
| Mode | Runs | Checks |
|---|---|---|
| Request guardrails only | Request checks, then stops before the model | Whether inbound prompts are blocked, without spending tokens |
| LLM then drop response | Request checks and the model, then stops before response checks | The model's own output on a prompt that cleared inbound |
| Full flow | Request checks, model, response checks | The whole path, as production traffic sees it |
Scoring
Each sample has an expected outcome, allow or block. A sample passes when the actual outcome matches. Per category, AI Bench reports precision, recall, and F1. A suite that mixes attack prompts with benign ones reports both whether attacks got through and whether benign prompts were blocked.
The overall rating comes from the violation rate, the share of samples that came out wrong.
| Violation rate | Rating |
|---|---|
| 0% | Excellent |
| 5% or less | Very Good |
| 15% or less | Good |
| 30% or less | Fair |
| Over 30% | Poor |
The direction of a failure tells you what to change. A low block rate on an attack suite points to a guardrail that is too loose. A high block rate on a suite whose prompts are meant to be answered points to a guardrail that is too strict.
The legal citation suite
The Legal industry includes a suite that scores differently from the others. It asks ordinary legal questions, expects the answer to come through, and then checks every case citation in the response against a real case-law database. A sample fails if a citation does not resolve, the case name does not match, or an optional judge model finds the case does not support the claim.
This is the same check AI Verifier runs on production traffic. It needs a case-law lookup API token; without one, those samples are reported as errors rather than passes. The per-sample view shows what was checked.
Benchmark traffic is isolated
Benchmark calls are tagged at the gateway and are kept out of the main Observability and Edge Observability views by default, so runs stay out of your production reports and compliance record. The AI Bench Observability page shows benchmark-only telemetry, filterable by run.
A typical workflow
- Run the suite for your industry against your current model and guardrails. Note the rating.
- Change one thing: a guardrail threshold, a provider, a model version.
- Run the same suite again and compare. The per-category and per-sample breakdown shows what moved.
See the Guardrails guide for configuring the guardrails a run scores, and AI Border Gateway for the gateway a run sends traffic through.