AI Border Gateway
AI Border Gateway sits between everyone in your organization calling an LLM or an MCP tool (developers, internal apps, agents) and the models and tools they call. Every call passes through it, tied to the same identity, routing, cost, and security checks.
The problem it solves
Once more than one team is calling LLMs, the same questions come up: who is calling which model, at what cost, and is anything they send or receive a compliance risk. Without a single point where every call passes through, answering those means checking logs in several different places, if they exist at all.
How it works
Every request is tied to an identity (a person, an app, or a team) before anything else happens. From there, routing picks which configured model handles the request; guardrails and budgets run before the call reaches a model and again on the response; every step is logged the same way, in one observability view for every model and every MCP tool call.
What's included
For the full capability reference, see AI Border Gateway capabilities.
Routing
- Register any number of LLM providers and models. A request names a model, and Cortega resolves it to a configured model behind the scenes.
- Health-based failover. Cortega tracks each model's error rate and latency with an exponentially weighted moving average, stops sending traffic to a model that's failing or slow until it recovers, and exposes the same metrics through Prometheus.
- Same-request retry. A failed call retries automatically against a configured set of retryable errors, with backoff, before the caller sees a failure.
- Virtual models. Group several real models behind one name, so callers don't need to know which model actually answered.
- Model Intelligence. Recommends which model to use, using your own traffic and usage, and names a specific replacement when a model you run gets deprecated. See Model Intelligence.
Cost management
- A cost and rate catalog per model, refreshed on a schedule.
- Budgets and spend limits, enforced before a call is allowed through.
Security
- Guardrails. Regex, webhook, or PII-detection (Presidio) checks, with built-in detectors for SSNs, credit cards, and emails, plus custom patterns. Apply globally or bind to a specific key.
- MCP tool governance. Grant a team access to specific MCP servers and, within a server, specific tools, checked on every tool call.
Applications
Tag traffic by which internal application sent it, using a request header you choose, so usage and cost break down per app, not just per team.
Identity, RBAC, observability, and audit logging are shared across every Cortega component, not specific to AI Border Gateway: see Platform.
Migrating from an existing LiteLLM gateway
A CLI tool, gwmigrate, extracts a running LiteLLM gateway's providers,
models, teams, guardrails, MCP servers, and virtual keys into a file. A
second command reviews that file against Cortega's current state and
creates the objects one approved step at a time, with rollback. See the
Gateway migration guide in Guides.
Scale and reliability
Cortega separates the control plane (the management backend you configure through the console) from the data plane (the gateway instances that carry traffic). Configuration is delivered to each gateway instance and applied locally, so AI Border Gateway keeps serving traffic from the last-delivered configuration even if the control plane is briefly unreachable. Run more than one gateway instance, optionally behind a load balancer: see the multi-node topology in AWS install.
Performance
Cortega's data plane is agentgateway, an open-source CNCF-aligned proxy. One 8 vCPU gateway instance sustained more than 52,000 requests per second at 100% success, adding a few milliseconds per call.
One gateway, under load
Measured on a single gateway instance, isolating gateway overhead by holding upstream latency constant (a mock upstream with a fixed 60 ms delay). Each row ran for 30 seconds at a fixed request rate; overhead is the median end-to-end latency minus that 60 ms.
| vCPU | Requests/sec | End-to-end p50 | Gateway overhead | Success |
|---|---|---|---|---|
| 2 | 5,000 | 61.7 ms | ~2 ms | 100% |
| 2 | 8,000 | 61.9 ms | ~2 ms | 100% |
| 2 | 12,000 | 64.8 ms | ~5 ms | 100% |
| 4 | 15,000 | 60.6 ms | ~1 ms | 100% |
| 4 | 25,000 | 61.9 ms | ~2 ms | 100% |
| 8 | 40,000 | 62.0 ms | ~2 ms | 100% |
| 8 | 48,000 | 62.7 ms | ~3 ms | 100% |
| 8 | 52,000 | 64.9 ms | ~5 ms | 100% |
| 8 | 53,000 | 69.3 ms | out of headroom | 99.97% |
Overhead stayed near 2 ms up to 40,000 requests per second and rose to about 5 ms as the 8 vCPU box approached its ceiling at 53,000. Throughput scaled with cores: doubling from 4 to 8 vCPUs roughly doubled the rate the gateway could carry. Streaming responses pass through token by token with no added buffering.
How it compares to LiteLLM
The agentgateway project benchmarked agentgateway against LiteLLM on identical hardware (Fortio load generator, mock backend, 32 connections, June and August 2026). LiteLLM's Rust mode measured slower than its Python mode.
| Gateway | Requests/sec | p99 latency added | Memory |
|---|---|---|---|
| Cortega data plane (agentgateway) | ~36,000 | ~2 ms | ~25 MB |
| LiteLLM, Rust mode | 983 | 71 ms | 2.15 GB |
| LiteLLM, Python mode | 3,198 | 32 ms | 11.8 GB |
LiteLLM publishes a 0.66 ms p99 figure for its Rust gateway. Benchmarked independently on that build, it sustained 983 requests per second at 71 ms p99. Source: agentgateway.dev.
Full methodology, our AIGatewayBench latency run, cost-per-million-requests figures, and updates as they're re-measured: Performance statistics.
Install
AI Border Gateway ships as one package, run with Docker Compose. See Install for the AWS, Terraform, and bring-your-own-infra paths.