AI Bench
AI Bench sends a set of sample prompts through your own gateway and model, with your current guardrails active, and scores what came back against the expected outcome. It gives you a measured pass rate for a given model and guardrail configuration, and a per-sample record of what passed and what did not.
AI Bench is its own console section, visible when your edition includes the
benchmarks feature. Viewing runs and starting them are separate
permissions.
How a run works
Cortega sends the prompts itself, through a gateway and model you select, using an internal shared credential. Production traffic is not involved.
The run does not create, change, or delete guardrails. It records the guardrails that are active at the time of the run and scores against that snapshot.
Every sample is a real governed call, so a run also surfaces connectivity and routing problems: a wrong provider key or an unreachable model shows up as errored samples.
Industries and suites
Each run is scoped to an industry, which selects the sample set.
| Industry | What it tests |
|---|---|
| General safety | Cross-policy safety categories: hate, harassment, self-harm, violence, sexual content, and their more severe variants |
| Healthcare | Triage boundaries, medication safety, PHI handling, emergency escalation |
| Finance | Unqualified financial advice, fraud enablement, market manipulation, financial PII |
| Legal | Legal-advice boundaries, confidentiality, evidence handling, and whether the model's own case-law citations are real (see below) |
| Privacy / PII | Whether the model discloses synthetic PII and PHI it should redact, across identity, contact, credential, and health or financial entities |
| Security / red-team | Jailbreak templates, prompt injection, credential abuse, cyber-harm behaviors, multi-turn and many-shot LLM suites, plus MCP Benchmark tool-output injection |
An industry can have more than one suite; the run form picks one. Each suite is a curated set of sample prompts, built and refined over time from public and research safety benchmarks.
Security / red-team currently ships three LLM suites:
- Security / Red-Team — single-request prompts (jailbreak templates, injection, credential abuse, cyber harm, over-refusal).
- Security / Red-Team Multi-Turn — a scripted conversation. Each turn is a separate governed request. Early setup turns must pass; later escalation, planted backdoors, or injected tool output must be blocked. The detail view shows the per-turn transcript. Techniques in this suite include Crescendo gradual escalation, PAIR-style refusal repair, TAP-style sibling rewrites, and AgentDojo-style email/payments/travel tool-output injection.
- Security / Red-Team Many-Shot — one long request, not a conversation. The prompt is packed with many in-context examples where an assistant appears to comply with unsafe asks, then a harmful target (Anthropic many-shot jailbreaking). Shot count is the independent variable (8/16/32/64). Request-guardrail mode is the intended scoring path: the classifier should catch the packed prompt before the model runs. Benign many-shot controls check that long defensive context is not over-refused.
Sample results record the technique and, for many-shot, the shot count.
MCP Benchmark
MCP Benchmark is a separate AI Bench item. Each sample is a user goal plus MCP tool results. The attack is not in the user prompt. It is in a field of a tool result (an email body, a payment memo, a hotel description, a ticket).
Cortega originates the probe, wraps the poisoned tool result in an MCP JSON-RPC envelope, and sends it through your gateway with guardrails on. A sample passes when the poisoned result is blocked. If it is allowed through, the run also records whether the model then proposed the injected tool call.
Request-guardrail mode is the intended first scoring path. Full flow adds the "did the agent follow the injection?" check.
Modes
The mode sets how far down the request and response pipeline each sample runs.
| Mode | Runs | Checks |
|---|---|---|
| Request guardrails only | Request checks, then stops before the model | Whether inbound prompts are blocked, without spending tokens |
| LLM then drop response | Request checks and the model, then stops before response checks | The model's own output on a prompt that cleared inbound |
| Full flow | Request checks, model, response checks | The whole path, as production traffic sees it |
Scoring
Each sample has an expected outcome, allow or block. A sample passes when the actual outcome matches. Per category, AI Bench reports precision, recall, and F1. A suite that mixes attack prompts with benign ones reports both whether attacks got through and whether benign prompts were blocked.
The overall rating comes from the violation rate, the share of samples that came out wrong.
| Violation rate | Rating |
|---|---|
| 0% | Excellent |
| 5% or less | Very Good |
| 15% or less | Good |
| 30% or less | Fair |
| Over 30% | Poor |
The direction of a failure tells you what to change. A low block rate on an attack suite points to a guardrail that is too loose. A high block rate on a suite whose prompts are meant to be answered points to a guardrail that is too strict.
The legal citation suite
The Legal industry includes a suite that scores differently from the others. It asks ordinary legal questions, expects the answer to come through, and then checks every case citation in the response against a real case-law database. A sample fails if a citation does not resolve, the case name does not match, or an optional judge model finds the case does not support the claim.
This is the same check Legal Agent runs on production traffic. It needs a case-law lookup API token; without one, those samples are reported as errors rather than passes. The per-sample view shows what was checked.
Benchmark traffic is isolated
Benchmark calls are tagged at the gateway and are kept out of the main Observability and Edge Observability views by default, so runs stay out of your production reports and compliance record. The AI Bench Observability page shows benchmark-only telemetry, filterable by run.
A typical workflow
- Run the suite for your industry against your current model and guardrails. Note the rating.
- Change one thing: a guardrail threshold, a provider, a model version.
- Run the same suite again and compare. The per-category and per-sample breakdown shows what moved.
See the Guardrails guide for configuring the guardrails a run scores, and AI Border Gateway for the gateway a run sends traffic through.