Series: Running a Subscription Business · Part 1 of 4
- Peak Trading Readiness: How We Prepare a Subscription Platform for Valentine's Day
- Involuntary Churn: The Engineering Behind Recovering Failed Subscription Payments
- SCA, 3D Secure 2 and Recurring Payments: An Architect's Guide for Subscription Platforms
- Winning the Pennies Unsung Hero Award: What Micro-Donations Mean in Tech
Contents
Peak Is a Product, Not a Date
For a flower business, Valentine's Day is the Black Friday of the year. The cost of an outage is not just lost revenue: it is a missed delivery on the one day it really mattered to someone.
With Valentine's Day a week away, I want to write down the readiness playbook we use. None of it is exotic. Readiness is not a heroic final week; it is a programme of work that starts two or three months out.
Start with a Traffic Model
Before you load test anything, you need a number to test against. I build a simple traffic model from last year's peak, adjusted for marketing plans and subscriber growth.
The important numbers are not daily totals. They are the peak minute rates for the handful of journeys that matter:
- Product and landing pages (mostly cacheable)
- Sign-up and checkout (not cacheable, touches payments)
- Account management (skip, pause, change delivery)
- Background work: order generation, charges, emails
Campaign emails create sharp spikes, not smooth curves, so get the send schedule from marketing and put it in the model.
Then apply a safety factor. I test to at least 2x the modelled peak, because models are wrong and you want to know where the ceiling is, not just that you cleared the floor.
Load Testing with k6
My default tool for this is k6. It is scriptable in JavaScript, runs well in CI, and its arrival-rate executors let you model requests per second rather than a fixed number of virtual users, which is closer to how real traffic behaves.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
valentines_peak: {
executor: 'ramping-arrival-rate',
startRate: 10,
timeUnit: '1s',
preAllocatedVUs: 200,
maxVUs: 1000,
stages: [
{ target: 60, duration: '5m' },
{ target: 150, duration: '10m' },
{ target: 150, duration: '20m' },
{ target: 0, duration: '5m' },
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<800'],
'http_req_duration{name:checkout}': ['p(95)<1500'],
},
};
const BASE_URL = __ENV.BASE_URL;
export default function () {
const products = http.get(`${BASE_URL}/api/products`, {
tags: { name: 'products' },
});
check(products, { 'products 200': (r) => r.status === 200 });
const checkout = http.post(
`${BASE_URL}/api/checkout/quote`,
JSON.stringify({ productId: 'demo-bouquet', postcode: 'SW1A 1AA' }),
{ headers: { 'Content-Type': 'application/json' }, tags: { name: 'checkout' } },
);
check(checkout, { 'quote 200': (r) => r.status === 200 });
}A few rules that save pain:
- Test in a production-like environment. Same instance classes, same database size, same queue and Lambda configuration. A test against an undersized staging environment tells you very little.
- Stub payments, don't skip them. Point the payment integration at the provider's test mode or a stub with realistic latency. A load test that removes the slowest dependency flatters you.
- Watch the whole system, not the k6 summary. The interesting failures show up in database connections, queue depth and third-party rate limits, not in the HTTP percentiles.
- Warn your providers. Some third parties rate-limit aggressively. Find out their limits before your test finds them for you.
Capacity Planning with Managed AWS Services
Managed services remove a lot of capacity work, but they do not remove all of it. The limits just move somewhere less obvious.
| Component | What to check | Typical action |
|---|---|---|
| Lambda | Account concurrency quota, reserved concurrency per function | Raise the quota well ahead; reserve concurrency for critical workers |
| SQS | Consumer throughput, visibility timeout vs processing time | Tune batch size; alarm on age of oldest message |
| Managed database | Connection limits, CPU, IOPS, replica lag | Scale up before peak; use a connection proxy for Lambda |
| CDN | Cache hit ratio on catalogue pages | Pre-warm and extend TTLs for static content |
| Service quotas | Any AWS limit you have never hit before | Request increases weeks, not days, before peak |
| Third parties | Email, SMS, address lookup and payment rate limits | Confirm limits in writing; queue anything that can wait |
Two points deserve emphasis. First, Lambda scales fast enough to overwhelm the things behind it. A database can fall over because a burst of workers all opened connections at once, so reserved concurrency is as much a brake as a guarantee. Second, AWS quota increases are not always instant. Raise them early.
For the database, I prefer to scale up vertically ahead of peak and scale back down afterwards. It is boring, predictable, and cheaper than an incident.
Change Freeze: Firm but Honest
We run a change freeze across the peak window. Freezes get a bad name because they are often vague. Ours has clear rules:
- Dates are published weeks in advance, so product teams plan around them.
- Only fixes for customer-affecting issues ship, with a named approver.
- Infrastructure changes are frozen too, including "harmless" configuration tweaks.
- Feature flag changes are allowed, because that is how we reduce risk during the freeze.
Kill Switches and Graceful Degradation
Every non-essential feature that runs in a critical path should have a kill switch: a feature flag that turns it off instantly without a deploy. We use LaunchDarkly for operational flags like these.
Before peak, we go through the critical journeys and ask one question for each dependency: if this is slow or down, what should the customer see?
- Recommendations or upsells slow? Hide the panel and render the page without it.
- Address lookup failing? Fall back to manual address entry.
- Email provider degraded? Queue confirmations and send them when it recovers; the order still succeeds.
- Analytics and tracking scripts? Turn them off before they compete with checkout for resources.
The rule is simple: the ability to take an order and deliver it is sacred. Everything else can degrade. A kill switch that nobody has tested is not a kill switch, so we flip each one in a production-like environment before peak.
Runbooks and the War Room
A runbook should be short enough to use under pressure. For each likely failure mode, it answers: how do I recognise it, what do I do first, who do I call, and how do I know it's fixed.
For the peak window itself:
- A named incident lead per shift, with authority to make decisions without a committee.
- An on-call rota that covers the early mornings, because that's when campaign emails land and deliveries start.
- One shared channel for the war room, with a pinned dashboard link and the runbook index.
- Customer service in the loop. They see problems before your alarms do. Give them a direct line into the channel.
- Pre-agreed escalation paths to AWS support and key vendors, with account numbers and contacts written down.
The best war rooms I've been part of are quiet.
Advice by Stage
| Stage | Focus |
|---|---|
| Startup | Pick the one or two journeys that must not fail. Load test those. Write a one-page runbook. |
| Scaleup | Formal traffic model, 2x load tests, change freeze, kill switches on every non-essential dependency, staffed on-call. |
| Enterprise | Game days and chaos testing, vendor capacity commitments, cross-team incident command, automated rollback. |
What to Measure Afterwards
The week after peak is when you earn next year's peak. We run a blameless review whether or not anything went wrong, and look at:
- Actual vs modelled traffic, per journey and per peak minute. Update the model.
- Headroom used: how close each component came to its limit.
- Error budget spend: checkout and payment error rates against target.
- Queue behaviour: maximum age of oldest message and how long backlogs took to clear.
- Kill switches flipped, and whether they did what we expected.
- Customer contact volume and its reasons, which often reveal issues the metrics missed.
Then turn the findings into backlog items with owners, and schedule them before the next peak, not the week before it.
The Takeaway
Peak readiness is mostly unglamorous preparation: a traffic model, realistic load tests, capacity raised ahead of time, a clear freeze, tested kill switches and short runbooks. Protect the ability to take and fulfil orders above everything else, and let everything around it degrade gracefully. If the war room is boring, you did the work.
