Incident Management for Small Engineering Teams: A Practical Playbook

Why Small Teams Get Incidents Wrong

Most small engineering teams handle incidents the same way: someone notices something is broken, posts in a busy Slack channel, and whoever happens to be online piles in. Three people investigate the same log line, nobody tells customer service what's going on, and the fix goes out without anyone writing down what happened.

It works, until it doesn't. The failure mode isn't usually the technical fix. It's coordination: duplicated effort, silence towards the business, and the same incident recurring two months later because the follow-up actions evaporated.

You don't need an SRE department to fix this. You need a small amount of structure, agreed in advance, that people can follow at 2am without thinking. In a subscription business with hard peaks like Valentine's Day and Mother's Day, that structure matters more than any single piece of tooling.

Severity Levels: Keep Them Few and Concrete

Severity levels exist to answer one question quickly: how much should we disrupt people for this? If engineers argue about the severity during an incident, the definitions are too abstract.

I use three or four levels, defined by customer and business impact rather than technical symptoms:

Two rules make this work. First, anyone can declare an incident and pick a severity, and it's always acceptable to start high and downgrade. Second, severity can change during the incident; it isn't a verdict, it's a routing decision.

On-Call That Doesn't Burn People Out

On-call is where small teams quietly lose good engineers. The rota is too short, the alerts are noisy, and nobody is compensated for being woken up. The pattern I've seen work:

• A rota of at least five or six people. Fewer than that and on-call comes round too often. If you don't have enough engineers, narrow what pages out of hours rather than shortening the rota.
• Page only on customer impact. Alert on symptoms (checkout error rate, payment failures, queue depth growing) rather than causes (CPU at 80%). Every page that didn't need a human is a tax on sleep.
• Review every out-of-hours page. A weekly five-minute look at what paged and why is the single best tool for cutting noise. If an alert fired and nobody needed to act, fix or delete it.
• Compensate it. A flat weekly allowance plus time off in lieu after a disturbed night. It's a small cost against the retention risk.
• Clear handovers. A short written handover at rota change: open issues, risky deploys, anything scheduled.
• Heightened cover for peaks. For known trading peaks, run a secondary on-call and freeze non-essential changes, rather than asking the normal rota to absorb the risk.

Asynchronous design helps too: as I described in "Queue Everything", when a downstream provider fails, messages wait in the queue rather than failing customer requests, so many problems become working-hours problems instead of 3am ones.

The Incident Commander Role

The most valuable change for a small team is naming one person as incident commander (IC) for every SEV1 and SEV2. The IC does not debug. Their job is to:

1. Open the incident channel and state the known impact.
2. Assign roles: who investigates, who communicates.
3. Keep a running timeline of decisions and actions.
4. Decide when to escalate, roll back or bring in a third party.
5. Declare the incident resolved and schedule the postmortem.

In a small team the IC is often an engineering manager or senior engineer, and for a SEV1 it may be me. The discipline is that the most technically capable person in the room usually should not be IC, because you want them debugging. Rotate the role and practise it on SEV2s so it isn't a novelty during a real outage.

Status Communication to the Business

Engineers underestimate how much damage silence does. Customer service is fielding complaints, marketing is about to send an email campaign into a broken checkout, and leadership is hearing about the problem second-hand.

Set a cadence and a template. For a SEV1, an update every 30 minutes even when there's nothing new; "still investigating, next update at 14:30" is useful information. Updates go to a single business-facing channel, not the engineering incident channel.

[SEV1] Checkout payments failing - update 3 (14:00)
Impact: New orders cannot be paid for. Existing subscriptions unaffected.
Status: Cause identified (payment provider timeout). Mitigation deploying.
Actions for you: Customer service - use the holding reply. Marketing - pause today's send.
Next update: 14:30 or sooner if resolved.

Write for a non-technical reader: impact, status, what they should do, and when they'll hear next. Keep the technical detail for the postmortem.

Blameless Postmortems

Every SEV1 and SEV2 gets a written postmortem within five working days. Blameless doesn't mean nobody is accountable; it means we assume people acted sensibly given what they knew, and we look for the system conditions that made the failure possible. If the answer is "an engineer ran the wrong command", the real question is why the wrong command was so easy to run.

A structure that works for small teams:

• Summary: what happened, impact and duration, in three sentences.
• Timeline: taken from the IC's notes, with timestamps.
• Contributing factors: usually several, rarely one root cause.
• What went well: detection, communication, tooling that helped.
• Actions: each with an owner and a due date.

Hold a 30-minute review meeting and invite people outside engineering when they were affected. It builds trust when the business sees the process is honest.

Action-Item Follow-Through

This is where most incident processes quietly fail. The postmortem is written, everyone nods, and the actions sit in a document nobody opens again.

What fixes it is boring: actions go into the normal backlog as tickets with the incident label, they're reviewed in a fortnightly reliability slot, and I report open postmortem actions alongside other engineering health indicators. Be selective; three actions that will get done beat ten aspirational ones. If an action isn't worth prioritising, close it explicitly and record why.

Tooling by Stage

Tooling should follow process, not replace it. A good tool makes the process easy to follow; it won't create one.

A basic Slack workflow gets a startup surprisingly far: a shortcut that asks for a severity and summary, creates a channel with a predictable name, posts the template and notifies the business channel. The moment you find yourself manually reconstructing timelines for postmortems, a dedicated tool like incident.io starts paying for itself.

The Takeaway

Good incident management in a small team is mostly about agreeing things in advance: what counts as serious, who is in charge, how the business hears about it and how lessons get turned into work. Keep severity levels concrete, protect on-call engineers from noise, separate the commander from the debugger and treat postmortem actions as real backlog items. Tooling helps, but only once the habits exist.