OpenTelemetry in Practice: Instrument Once, Switch Observability Vendors with Config

Where Lock-In Actually Lives

When a team wants to move from one observability vendor to another, the dashboards and alerts are annoying to rebuild but manageable. The real cost is in the code: vendor-specific APM agents, custom metrics libraries and tracing SDKs in every service. Switching becomes a multi-quarter re-instrumentation project, and the vendor knows it at renewal time.

OpenTelemetry (OTel) is the CNCF standard for generating and exporting traces, metrics and logs. Every major vendor accepts it, and AWS, Google and Azure all support it natively. Instrument with OTel, and the choice of backend becomes configuration.

The Architecture

 Services (OTel SDK + auto-instrumentation)
        │  OTLP
        ▼
 OTel Collector  ── receivers → processors → exporters ──┬─► CloudWatch / X-Ray
 (agent or gateway)                                      ├─► Grafana (Tempo, Mimir, Loki)
                                                         └─► Datadog / Honeycomb / New Relic

SDKs and auto-instrumentation in each service produce telemetry in a vendor-neutral format (OTLP). For Node.js, Java, Python and .NET, auto-instrumentation covers HTTP servers and clients, database drivers, AWS SDK calls and message queues with almost no code.
The Collector receives OTLP, processes it (batching, sampling, attribute redaction, enrichment) and exports to one or more backends.

The Collector is where vendor decisions live. Changing vendors, or sending to two during a migration, is a change to one YAML file.

Instrumenting a Node.js Service

// instrumentation.ts: load before the app starts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
 
const sdk = new NodeSDK({
  serviceName: process.env.OTEL_SERVICE_NAME ?? "orders-api",
  traceExporter: new OTLPTraceExporter(),          // defaults to the local collector
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
 
sdk.start();

Add business context where it matters, using the vendor-neutral API:

import { trace, metrics } from "@opentelemetry/api";
 
const ordersPlaced = metrics.getMeter("checkout").createCounter("orders_placed");
 
export async function placeOrder(order: Order) {
  const span = trace.getActiveSpan();
  span?.setAttribute("order.market", order.market);
  span?.setAttribute("order.plan", order.plan);
  // ...
  ordersPlaced.add(1, { market: order.market });
}

Configure the endpoint and resource attributes with standard environment variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production) so the same build runs everywhere.

The Collector Config That Makes Switching Easy

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
  batch:
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
 
exporters:
  awsxray:
  awsemf:
    namespace: Services
  otlphttp/grafana:
    endpoint: https://otlp-gateway.example.grafana.net/otlp
    headers:
      authorization: "Basic ${env:GRAFANA_OTLP_TOKEN}"
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/redact, tail_sampling, batch]
      exporters: [awsxray, otlphttp/grafana]     # dual-write during a migration
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awsemf]

Three things in that file do most of the work:

Redaction at the collector. Sensitive attributes never leave your network, regardless of vendor.
Tail sampling. Keep every error and every slow trace, sample the rest. This is the single biggest lever on observability cost.
Multiple exporters. Dual-write for a few weeks when evaluating or migrating vendors, compare, then remove the old one.

On AWS, the AWS Distro for OpenTelemetry (ADOT) packages the collector with the AWS exporters, and it runs as a Lambda layer, an ECS sidecar or a Kubernetes DaemonSet. Tail sampling needs all spans of a trace to reach the same collector, so for that you'll run a gateway tier (with load balancing by trace ID) behind the per-node agents.

Deployment Patterns

Pattern	How	When
Direct export	SDK exports straight to the vendor	Prototypes only; puts vendor config in every service
Agent	Collector per node/task, services send to localhost	Default for most teams
Agent + gateway	Agents forward to a central collector tier	Tail sampling, central redaction, multiple backends

Adopting It Without a Big Bang

Start with traces on your most important request path (for us, checkout and subscription changes). Auto-instrumentation gets you 80% of the value on day one.
Standardise resource attributes: service.name, service.version, deployment.environment.name. Consistent naming is what makes cross-service views work.
Propagate context through queues. Auto-instrumentation handles HTTP; for SQS and event buses, make sure trace context travels in message attributes so async work joins the original trace.
Move metrics next, keeping existing dashboards alive by exporting to the current backend.
Logs last: add trace IDs to your structured logs first, which delivers most of the correlation value before you move log pipelines.

The Takeaway

OpenTelemetry doesn't make observability cheaper by itself, but it moves the decision about where telemetry goes out of your application code and into a collector you control. That gives you redaction and sampling in one place, the freedom to dual-write while you evaluate, and a much stronger negotiating position at renewal. Instrument once; choose vendors with configuration.