CloudWatch as Your Observability Backbone: Native Dashboards, Grafana, or Something Else?

CloudWatch Is Already There

Every Lambda invocation, every RDS instance, every load balancer and SQS queue you run on AWS is already publishing metrics to CloudWatch. Your Lambda logs are already in CloudWatch Logs. Before you evaluate any observability vendor, you should recognise you already have a data source that costs nothing to set up and covers most of your infrastructure.

In 2018 I wrote about running Grafana and New Relic on our Kubernetes estate. Much has changed since then, but the core lesson hasn't: observability is about answering questions quickly during an incident, not about how pretty the dashboards are.

What CloudWatch Gives You

Metrics. AWS service metrics for free (with standard resolution), plus custom metrics you publish. Embedded Metric Format (EMF) lets you write a structured log line and have CloudWatch extract metrics from it, which is the cheapest way to get business metrics out of a Lambda:

console.log(JSON.stringify({
  _aws: {
    Timestamp: Date.now(),
    CloudWatchMetrics: [{
      Namespace: "Checkout",
      Dimensions: [["Market"]],
      Metrics: [{ Name: "OrdersPlaced", Unit: "Count" }],
    }],
  },
  Market: "UK",
  OrdersPlaced: 1,
  orderId: "ord_123",
}));

Logs and Logs Insights. A query language that's good enough for most incident investigations:

fields @timestamp, @message
| filter level = "error" and service = "payments"
| stats count() as errors by bin(5m) as window, errorCode
| sort window desc

Alarms. Threshold, anomaly detection and composite alarms, routed via SNS to Slack, PagerDuty or email.

Dashboards. Functional, shareable, and able to mix metrics and Logs Insights widgets.

Application Signals, X-Ray and Container Insights. Tracing, service maps and Kubernetes/ECS metrics, with OpenTelemetry support, so you're not locked into a proprietary SDK.

CloudWatch as an event source. Alarm state changes are published to EventBridge. That means an alarm can trigger automation: scale something, open a ticket, post a rich Slack message, or run a Lambda that captures diagnostics at the moment of failure.

Where CloudWatch Frustrates

• Dashboards are functional, not delightful. Building good ones takes effort, and templating across environments is clumsy unless you define them in Terraform or CDK.
• Cross-account and cross-region views need deliberate setup (cross-account observability works, but you have to configure it).
• Costs hide in the details. Log ingestion per GB, custom metrics per metric per month, high-cardinality dimensions and Logs Insights per GB scanned. A chatty debug log left on in production can quietly become one of your larger AWS line items.
• It's AWS-only. If you have significant workloads elsewhere, or third-party services you care about, you'll want one pane of glass.

Three Ways to Build on It

Option 1: Native CloudWatch Only

Dashboards, alarms and Logs Insights, all in AWS.

Best for: startups on a tight budget with an AWS-only, mostly serverless stack. You pay only for what you ingest and query, and there's no new vendor, contract or security review.

Make it work: define dashboards and alarms in code, standardise structured JSON logging from day one, set log retention on every log group (the default is never expire), and publish a handful of business metrics via EMF so dashboards show orders and sign-ups, not just CPU.

Option 2: CloudWatch as a Data Source for Grafana

Grafana (self-hosted OSS, Grafana Cloud, or Amazon Managed Grafana) reads directly from CloudWatch metrics and Logs Insights. Your data stays in CloudWatch; Grafana becomes the presentation and alerting layer.

Best for: teams that want better dashboards, templating (one dashboard, a dropdown for environment or service), and the ability to combine CloudWatch with other sources: Prometheus from a Kubernetes cluster, Postgres for business data, or Loki for logs.

Watch out for: Grafana queries CloudWatch APIs, and GetMetricData calls are billed. A dashboard with 40 panels auto-refreshing every 10 seconds on a TV in the office will cost you real money. Set sensible refresh intervals.

Self-hosted vs managed: self-hosted Grafana OSS is free and runs happily on a small container, but you own upgrades, auth and availability (and your monitoring must not go down with the thing it monitors). Grafana Cloud's free tier is generous for small teams. Amazon Managed Grafana integrates with IAM Identity Center and is priced per active user, which is excellent for small engineering teams and less so if you want the whole company to have access.

Option 3: A Third-Party Platform (Datadog, New Relic, Honeycomb, etc.)

Ship metrics, logs and traces to a vendor that does correlation, APM, anomaly detection and polished UX out of the box.

Best for: scaleups and enterprises where engineer time during incidents is the most expensive resource, and multi-cloud or hybrid estates.

Watch out for: pricing that scales with hosts, containers, custom metrics, log volume and users all at once. Every observability vendor bill I've reviewed had at least one line nobody understood. Negotiate committed-use contracts, sample traces, and filter logs before they're shipped.

Startup vs Enterprise

The Principle That Saves You Money: Instrument with OpenTelemetry

Whatever you choose today, instrument your applications with OpenTelemetry rather than a vendor SDK. OTel lets you send the same telemetry to CloudWatch, Grafana (Tempo/Loki/Mimir), Datadog or Honeycomb by changing collector configuration rather than application code. That turns your observability vendor into a procurement decision rather than a re-instrumentation project, and gives you leverage when renewal comes around.

What to Put on the First Dashboard

Resist dashboards with 60 panels. For each service, start with:

1. Request rate, error rate and latency (p50/p95/p99): the RED metrics.
2. Queue depth and age of oldest message for every SQS queue. Age of oldest message is the single most useful async metric I know.
3. DLQ message count, with an alarm on anything above zero.
4. Database connections, CPU and replication lag.
5. Two or three business metrics: orders per minute, sign-ups, payments succeeded vs failed. When the business line drops and the technical lines look fine, you've still got an incident.

The Takeaway

For a startup on AWS, native CloudWatch with disciplined structured logging and a few business metrics will take you a surprisingly long way for very little money. Add Grafana on top when you need better dashboards or a second data source. Move to a commercial platform when incident time costs more than the invoice. And instrument with OpenTelemetry so that moving between these options is a configuration change, not a rewrite.