Deploying an LLM Gateway with LiteLLM: Virtual Keys, Budgets and Routing Claude Code Through It

What We're Building

An internal LLM gateway that every AI request in the company passes through, whether it's a developer's coding agent, an internal app or a CI bot. The goals:

• No provider API keys on laptops or in application config.
• A virtual key per team or service, with budgets and model allow-lists.
• Spend attributed to teams, visible to finance.
• Sensitive data redacted before it leaves the network.
• Provider fallback, so one provider's outage doesn't stop engineering.

LiteLLM is the most widely used open-source option for this. It exposes OpenAI-compatible and Anthropic-compatible endpoints, and routes to 100+ providers, including the Anthropic API, Amazon Bedrock, Vertex AI, Azure OpenAI and self-hosted models.

Architecture on AWS

 Developers / apps / CI
        │  HTTPS (virtual key)
        ▼
 Internal ALB (private, company VPN / SSO-protected)
        ▼
 ECS Fargate service: LiteLLM proxy (2+ tasks)
   ├── RDS Postgres  → keys, teams, budgets, spend logs
   ├── ElastiCache Redis → rate limits, cross-instance budget tracking
   ├── Secrets Manager → provider credentials
   └── Egress → Anthropic API · Bedrock (via VPC endpoint) · other providers

Run at least two tasks across availability zones. The gateway is now on the critical path for every engineer using AI, so treat it like production infrastructure: health checks, autoscaling, alarms and a runbook.

The Configuration

A config.yaml that defines logical model names, maps them to providers and sets gateway behaviour:

model_list:
  # Flagship model for complex coding work, with a Bedrock fallback
  - model_name: claude-flagship
    litellm_params:
      model: anthropic/<anthropic-model-id>
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-flagship-bedrock
    litellm_params:
      model: bedrock/<bedrock-model-id>
      aws_region_name: eu-west-2
 
  # Cheaper model for routine and high-volume tasks
  - model_name: claude-fast
    litellm_params:
      model: anthropic/<smaller-model-id>
      api_key: os.environ/ANTHROPIC_API_KEY
 
router_settings:
  num_retries: 2
  fallbacks:
    - claude-flagship: ["claude-flagship-bedrock"]
  redis_host: os.environ/REDIS_HOST
  redis_port: 6379
 
litellm_settings:
  drop_params: true
  # Log metadata only; decide deliberately before storing full prompts
  turn_off_message_logging: true
 
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

A few decisions worth calling out:

• Logical model names (claude-flagship, claude-fast) decouple clients from provider model IDs. When a new model ships, you change one line in the gateway rather than every client.
• Fallbacks to the same model family on another provider keep engineers working during an outage.
• Message logging off by default. Spend and usage metadata is enough for most purposes. If you need full prompt logs for debugging or audit, send them to storage with strict access controls and retention, and tell people you're doing it.

Teams, Virtual Keys and Budgets

Create a team with a budget and model allow-list, then issue keys under it:

# Create a team with a monthly budget
curl -X POST "$GATEWAY/team/new" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"team_alias": "platform", "models": ["claude-flagship", "claude-fast"],
       "max_budget": 1500, "budget_duration": "30d"}'
 
# Issue a key for an engineer (or a service) within that team
curl -X POST "$GATEWAY/key/generate" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"team_id": "<team-id>", "key_alias": "jane-laptop",
       "max_budget": 300, "budget_duration": "30d", "rpm_limit": 60}'

Now every request is attributed to a key and a team, budgets are enforced, and revoking one key doesn't affect anyone else. The admin UI shows spend by team, key and model, which becomes your monthly AI cost report. SSO for the admin UI is available, though some identity features depend on LiteLLM's licence tier, so check what your setup needs.

Guardrails: Redacting Before Data Leaves

LiteLLM supports guardrails that run before and after model calls, including integration with Microsoft Presidio for PII detection and masking. A sensible starting set:

• Secrets detection: block or mask API keys, private keys and connection strings in prompts. Coding agents read .env files more often than you'd think.
• PII masking: customer emails, phone numbers and card numbers.
• Model allow-lists per data classification: teams handling customer data can only use providers and regions approved for it.

Guardrails add latency, so apply heavier checks to the routes that need them rather than globally.

Pointing Claude Code at the Gateway

Claude Code can use a gateway via environment variables. Using the Anthropic-format endpoint keeps Claude-specific features working:

export ANTHROPIC_BASE_URL="https://llm-gateway.internal.example.com"
export ANTHROPIC_AUTH_TOKEN="sk-<virtual-key>"

For a company rollout, don't rely on engineers setting these by hand. Distribute them through Claude Code's managed settings, deployed with your device management tooling, so every machine uses the gateway by default. Other tools (Cursor, IDE plugins, internal apps using the OpenAI or Anthropic SDKs) are configured the same way: a base URL and a virtual key.

Test before rolling out. Streaming, tool use, prompt caching headers and newer API features need to pass through the gateway correctly, and gateway versions sometimes lag new provider features.

Operating It

• Dashboards: requests, error rates by provider, latency percentiles, spend by team, and budget-exceeded events.
• Alarms: provider error rates, fallback activations, and the database and Redis health checks.
• Upgrades: LiteLLM moves quickly. Pin versions, test in staging with real tools, and read release notes for breaking changes.
• Monthly review: spend per team and model, unused keys to revoke, and whether routine work could move to cheaper models.

Rollout Plan

1. Pilot with one team for two weeks. Measure latency overhead and fix tooling issues.
2. Issue keys through a self-service flow (a simple internal form or a Slack workflow) so the gateway doesn't become a ticket queue.
3. Migrate internal applications, replacing direct provider keys with virtual keys.
4. Rotate and revoke the old provider keys once traffic has moved. Until then, you haven't gained the control you set out to get.

The Takeaway

A self-hosted LLM gateway is a small amount of infrastructure that pays for itself the first time someone asks "what are we spending on AI, and where does our code go?" Keep the configuration simple, use logical model names, make key issuance self-service, and be deliberate about what you log. The gateway is where your AI policy stops being a document and starts being enforced.