Running an LLM Gateway in Your Business: Keeping Context and Code In-House (and How It Differs from Bedrock)

The Problem a Gateway Solves

In most companies, AI adoption starts organically: a few engineers with personal API keys, someone using a browser extension, a product team calling a model directly from a Lambda. Six months later you have:

• API keys scattered across laptops, CI secrets and .env files.
• No idea which teams are spending what.
• No consistent answer to "where does our source code and customer data go when someone uses AI?"
• Every application implementing its own retries, fallbacks and rate-limit handling.

An LLM gateway is a single service that sits between your people and applications and the model providers. Every AI request goes through it.

Developers (Claude Code, Cursor, IDE plugins)  ┐
Internal apps and agents                       ├─► LLM Gateway ─┬─► Anthropic API
CI / code review bots                          ┘   (your VPC)   ├─► Amazon Bedrock
                                                                ├─► Google Vertex AI
                                                                ├─► OpenAI / Azure OpenAI
                                                                └─► Self-hosted models

What a Gateway Gives You

Central credentials. Provider keys live in the gateway (backed by a secrets manager). Users and services get virtual keys issued by the gateway, scoped to a team, with budgets and model allow-lists. Revoke one person's access without rotating anything else.

Identity and attribution. Integrate with your SSO so every request is tied to a user and team. Now you can answer "who spent £4,000 on tokens last week?"

Policy enforcement. Which models are approved, for which teams, with which data classifications. Block unapproved providers entirely.

Data controls. Redact or block secrets and personal data before a prompt leaves your network. Detect API keys, card numbers and customer emails in prompts.

Logging and audit. A record of what was sent and received, stored in your account, with your retention rules. Decide deliberately whether to log full prompts (useful for debugging, sensitive to store) or metadata only.

Reliability. Retries, fallbacks between providers or regions, load balancing across multiple accounts, and rate limiting per team so one runaway agent doesn't exhaust everyone's quota.

Cost control. Budgets and alerts per team or key, caching where appropriate, and routing routine tasks to cheaper models.

Portability. Applications call one API; you can change the provider or model behind it without changing application code.

The Options

• Open-source, self-hosted: LiteLLM is the most common choice; others include Portkey's open-source gateway and Envoy-based AI gateways. You run it in your own VPC, on ECS, EKS or any Kubernetes cluster, in front of Postgres/Redis for state.
• Cloud-native: AI gateway features in existing API management products (Kong, Azure API Management, Cloudflare AI Gateway and others).
• Commercial SaaS gateways: quicker to start, but the gateway itself is now a third party in your data path, which partly defeats the point for sensitive use cases.

Keeping Context and Code In-House

"Keeping everything in the business" is the most common reason I hear for a gateway, and it's worth being precise about what a gateway can and can't do.

What stays in-house with a self-hosted gateway:

• Credentials and keys.
• Logs, prompts and responses you choose to keep, in your own storage and region.
• Usage data, cost data and audit trails.
• Your policy decisions: routing, redaction, allow-lists.
• Your context: the repository instruction files, specs, retrieval indexes and knowledge bases your tools use live in your repos and your infrastructure, and are sent per request, not uploaded to train anything.

What still leaves your network: the prompt itself, including any code or documents in the context, has to reach the model to be processed. A gateway governs what is sent and where; it can't make an external model process data it never receives. So the real controls are:

1. Choose providers and plans with the right terms. Business and API terms typically exclude your data from model training, but retention periods, zero-data-retention options and regional processing vary by provider, product and model. Read them for each model you approve.
2. Choose where inference runs. Route sensitive workloads to a provider path in your preferred region, or inside your cloud account (see Bedrock below).
3. Minimise and redact. Send only what's needed. Strip secrets and personal data at the gateway.
4. Self-host for the most sensitive cases. Open-weight models on your own GPUs keep everything in-house, at the cost of capability and operational effort.

Configuring Coding Tools to Use It

Most coding agents and SDKs can be pointed at a custom base URL. For example, Claude Code supports routing through a gateway via environment variables, which you can set centrally with managed settings:

# Route Claude Code through the company gateway
export ANTHROPIC_BASE_URL="https://llm-gateway.internal.example.com"
export ANTHROPIC_AUTH_TOKEN="<virtual key issued by the gateway>"

A LiteLLM configuration that exposes one logical model backed by two providers might look like:

model_list:
  - model_name: coding-default
    litellm_params:
      model: anthropic/<model-id>
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: coding-default
    litellm_params:
      model: bedrock/<bedrock-model-id>
      aws_region_name: eu-west-2
 
router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
 
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Check the current documentation for your gateway and tools; the configuration keys change between versions.

How This Is Different from Amazon Bedrock

This is the question I'm asked most, because Bedrock also sounds like "one place to access many models inside AWS". They're complementary layers, not alternatives.

Amazon Bedrock is a model provider platform. It gives you access to foundation models (including Anthropic's Claude models, Amazon's own models and several open-weight families) through AWS APIs, inside your AWS account's security boundary: IAM for auth, VPC endpoints (PrivateLink) so traffic doesn't traverse the public internet, CloudTrail for API audit, CloudWatch for metrics, AWS billing and your existing AWS commitments. It adds platform features like Guardrails, Knowledge Bases and Agents. For Claude specifically, Bedrock is operated by AWS with its own pricing, and new features can sometimes arrive on the first-party API before they're on Bedrock.

An LLM gateway is a control plane in front of one or more providers, which may include Bedrock.

When Bedrock alone is enough

If you're all-in on AWS, happy with the models it offers in your region, and your main concern is keeping inference inside your AWS boundary with IAM and CloudTrail, Bedrock on its own can be the whole answer. Many AI coding tools, Claude Code included, can be configured to use Bedrock directly. You get data-perimeter comfort without running anything.

When you want a gateway (with or without Bedrock)

• You use more than one provider (e.g. Claude via the Anthropic API for the newest features, Bedrock for workloads that must stay in AWS, and another vendor for specific tasks).
• You need per-user and per-team keys, budgets and attribution that finance and security can understand without reading IAM policies.
• You want one consistent policy layer for redaction, logging and model allow-lists across every tool and application.
• You want to change providers without touching application code.

The pattern I like for a mid-size business: gateway in your VPC → Bedrock (via PrivateLink) for sensitive, in-perimeter workloads, plus direct provider APIs for the latest models where the terms are acceptable. One entry point, one audit trail, and a deliberate choice of where each class of data goes.

By Stage

Startup: Probably no gateway yet. Use a business/team plan for your coding tools, and one provider account with separate keys per environment. If you're on AWS and data sensitivity matters, Bedrock is an easy default.

Scaleup: Introduce a self-hosted gateway once you have several teams and a meaningful AI bill. Virtual keys per team, SSO, budgets, basic redaction and metadata logging.

Enterprise: Gateway as a platform service owned by a platform or AI enablement team, integrated with DLP, SIEM and FinOps; approved-model catalogue per data classification; Bedrock or other in-cloud providers for regulated data; self-hosted models for the narrow set of cases where nothing may leave.

The Takeaway

A gateway doesn't stop prompts leaving your network, but it makes sure you decide, deliberately and visibly, what leaves, where it goes, who sent it and what it cost. Bedrock is one of the best places to send it if you live in AWS. The two work well together: Bedrock as a provider inside your cloud perimeter, the gateway as your company-wide control point across every provider you use.