The thing you wanted finally happened. The product got picked up — a TechCrunch hit, a tweet that went sideways, a feature on a podcast with real reach. Traffic went from two hundred concurrent users to eleven thousand in under four minutes. And then, at the exact moment more people were looking at your product than ever would again, it served a 502 to most of them. The spike you spent two years chasing arrived, and the system met it by folding.
Here's the part that stings: it almost certainly wasn't your app servers that broke. They're the part you can scale by adding boxes. The thing that broke was something downstream that couldn't scale by adding boxes, and every team learns which part that is the hard way — during the spike, in production, while the people you most wanted to impress watch a spinner.
You don't get to schedule the spike. You do get to decide, in advance, what happens when it comes. Let's walk the failure from front to back.
Where it actually breaks first
A load spike doesn't break your system evenly. It finds the single least-elastic component in the request path and breaks that, and then everything upstream of it backs up. The order is remarkably consistent.
The stateless app tier is rarely the first to go, because it's the easy part — it's horizontally scalable by design. Add more instances, put a load balancer in front, done. If your app servers were the bottleneck, you'd have a good problem.
The first thing to break is almost always something with a fixed pool: a connection limit, a thread pool, a rate-limited third-party dependency. Your app tier scales out to forty instances under autoscaling, and all forty open connections to a database that allows a hundred. Now you have a connection storm exhausting the database before the data load itself is even the issue. Or your checkout calls a payment provider that rate-limits you at fifty requests per second, and the spike sends five hundred, and four hundred fifty of them fail.
spike -> app tier autoscales 4 -> 40 instances (fine, elastic)
-> each opens 25 db connections = 1000 (db cap is 100)
-> connection storm -> db refuses -> 502s upstream
the app tier never broke. the thing it couldn't scale did.
The lesson: before a spike, find the fixed pools. Database connections, downstream rate limits, file handles, a single Redis instance, a payment gateway quota. Those are your real capacity, not your app-server count. The narrowest fixed pool in the path is your ceiling, and autoscaling the elastic tier just makes you hit it faster.
Autoscaling reality
Autoscaling is real and worth having, but the marketing version and the production version are different animals, and the gap between them is where launches die.
Autoscaling is reactive and slow. It watches a metric — CPU, request count — notices it crossed a threshold, then boots new instances. Booting an instance, pulling the image, warming the runtime, passing health checks, joining the load balancer: that's ninety seconds to several minutes on a good day. Your spike went 50x in four minutes. By the time the new capacity is ready, the surge has already either passed or already knocked you over. Reactive autoscaling handles a ramp; it does not handle a step function.
Two things make it actually work. Pre-warming: if you know the launch window — the press embargo lifts at 9am, the campaign goes out Tuesday — you scale up before, on a schedule, and ride into the spike with headroom already provisioned. The best capacity decision is the one you made an hour early. Scaling on the right signal: scaling on CPU when your bottleneck is the database connection pool is scaling the wrong tier. More app instances pointed at a saturated database make the outage worse, not better. Scale on the metric that reflects your actual constraint, or you'll faithfully autoscale yourself off a cliff.
And autoscaling has no answer at all for the fixed pools above. You cannot autoscale your way past a payment provider's rate limit. That requires the next two tools.
Queueing and load shedding: the two honest answers
When demand exceeds what the system can serve, there are exactly two honest responses, and a system needs both: defer the work, or refuse it. Pretending you can serve everything is the lie that produces 502s.
Queueing decouples accepting work from doing it. The spike hits, requests come in faster than you can process them, so you accept them onto a durable queue and process at a sustainable rate. This is how you absorb a burst without dropping it — the order gets placed instantly, the confirmation email and inventory reservation happen seconds later as the queue drains. Queueing turns a spike that would have crushed a synchronous path into a backlog that clears over minutes. It works only for work that can be asynchronous; you can't queue rendering the page the user is staring at.
Load shedding is the discipline of refusing work you can't serve, fast and cheaply, instead of accepting it and dying slowly. When the system is at capacity, an explicit "503, try again in a moment" returned in two milliseconds is infinitely better than a connection accepted, held, and timed out forty seconds later. The slow-timeout path is what actually kills you — it holds resources for doomed requests, so the requests you could have served get starved too. A system that sheds load at the edge stays up serving the traffic it can handle. A system that accepts everything serves nobody.
The combination: queue what can wait, shed what can't be served, and protect the core path so it keeps working for the requests that make it through.
The database bottleneck
The database deserves its own section because it's the most common single point of collapse and the hardest to fix mid-spike.
Reads are the usual culprit, and reads have a clean answer: a read replica or three takes the heavy list-views, dashboards, and search queries off the primary, and a CDN/edge cache in front means the public read traffic — the landing page everyone just clicked — never reaches your database at all. The launch traffic is overwhelmingly people looking, not people writing. If the thing everyone is hitting is served from cache at the edge, the spike never reaches the part that can't scale. That single move saves more launches than any other.
Writes are harder, because a primary can only accept so many. This is where queueing earns its place again: accept the write fast, queue it, persist at a sustainable rate. And it's where connection pooling is non-negotiable — PgBouncer in transaction mode so your forty app instances multiplex onto a few dozen real database connections instead of storming the connection cap. The connection storm from the first section is prevented entirely by a pooler the team usually skips because it's invisible until the day it isn't.
Graceful degradation: decide what stays up
The last decision is a product decision wearing an engineering costume. When the system is past capacity, what do users get? "Everything, slowly, until it crashes" is the default, and it's the worst option. The right answer is decided in advance: the core path stays up, the enhancements fall away.
Recommendations service overloaded? Serve the page without recommendations. Live inventory count under strain? Show a cached number and reconcile at checkout. Personalization layer struggling? Serve the generic version. The order is: keep the thing that makes you money working, drop the things that merely make it nicer. A user who can still buy, on a slightly plainer page, is a conversion. A user staring at a 502 is the press hit you wasted.
This requires having drawn the line before the spike — which features are core, which are enhancing, and what each one falls back to under load. You can't make that call in the middle of an incident.
What fixed looks like
The press hit lands. The public landing page everyone clicks is served from the CDN edge — your origin barely notices. Read replicas absorb the dashboard and search load; the primary handles writes through a connection pooler that keeps the connection count sane no matter how many app instances autoscaling spins up. You pre-warmed an hour before the embargo lifted, so capacity was already there, not three minutes behind the curve.
Async work — emails, inventory reservations, downstream syncs — rides a queue and drains over the next few minutes instead of fighting the surge synchronously. At the very edge, load shedding returns a fast, honest 503 to the small fraction of traffic above your real ceiling, so the requests you can serve stay fast. Non-essential features degrade quietly; checkout never does. The spike shows up as a tall, calm graph and a queue that drains, not a war room. The people you wanted to impress got a fast product.
This is for you if
You're a funded team with a launch, a campaign, or a growth curve that's about to send a step-function of traffic at a system that's only ever seen a steady trickle — and you'd rather find the bottleneck now than during the moment you've been working toward. You want to know your real ceiling before the spike tells you.
A load-readiness engagement runs $50k+: we find your fixed pools and your real ceiling, put caching and replicas where they belong, add queueing and load shedding on the paths that matter, and then load-test the system to failure on purpose so the first big spike isn't the first time it's seen one. A full scale-out re-architecture for a system expected to live through sustained viral or seasonal load runs $100k+.
It's not for you if you're pre-launch with no real traffic and no spike on the horizon — premature scaling is just complexity you'll pay to maintain for a load that isn't coming. It's for the team that can see the spike coming and refuses to meet it with a spinner.
// find the ceiling before the spike does
< transmit >