One Gateway Many Models

Design an LLM gateway. Every product team in the company calls one internal endpoint instead of each provider directly, and the gateway handles routing, streaming, caching, quotas, cost accounting, and failover.

Functional requirements

Accept a chat completion request naming a logical model such as "fast" or "deep" and route it to a concrete provider model. Stream tokens back. Enforce a per-team monthly spend cap and a per-key rate limit. Attribute every token to a team and a feature. Fail over when a provider degrades. Let a team pin a specific model version.

Non-functional requirements

The gateway must add under 30ms to time to first token at p95. It must never be the reason a request fails when any provider is healthy. Spend caps must hold under concurrency, since two simultaneous requests must not both pass a check that only one had budget for. No prompt content in logs by default.

Scale

60 teams, 400 API keys, 25,000 requests per minute at peak with 40% of them streaming for over 30 seconds. Average 12,000 input and 800 output tokens. Four providers, roughly 30 concrete models. Monthly spend is in the millions of dollars.

Out of scope

Building the models, the finance system the cost data feeds, and the developer portal.