Most SaaS doesn't die from traffic spikes. It dies from a schema decision made in week two, a retry loop nobody tested, and a cascade that starts with a single failed job. The traffic spike is the proximate cause. Everything else was the setup.
Post-mortems that should have been pre-mortems.
The four categories of production failure recur with enough consistency to constitute a taxonomy. They're not random. They're the predictable consequences of specific decisions made early, under conditions where the consequences weren't visible yet. The way to avoid them is to recognize the pattern before you're in it.
Category 1: Data model constraints
The failure mechanism
The schema was designed for the current feature set, not the domain. As the product evolves, the application code compensates: nullable columns accumulate, application-level constraints replace database-level constraints, relationships that should be foreign keys are managed in code.
At some point — usually triggered by a bug fix or a new feature — the application code that manages these implicit constraints gets out of sync with the data. Records exist in states the domain shouldn't allow. The application assumes invariants that the database doesn't enforce.
The failure presents as data corruption, not a crash. Users report incorrect results. Investigation reveals records in impossible states. The fix requires a data migration that has to reconstruct valid state from invalid data — often with judgment calls that require business context.
The specific pattern
A SaaS platform has a subscriptions table with a status column managed entirely in application code. Legitimate statuses are active, trialing, canceled. Over 18 months, events like failed webhook deliveries, race conditions in async jobs, and a bug that was fixed by rerunning the affected records leave 3.4% of subscriptions in states that the application's status machine didn't anticipate — either null, or a string from a previous version of the status logic.
The failure surfaces when a billing engineer adds a query that filters on status = 'active' and the query behaves unexpectedly because the implicit semantics of active have drifted from the actual data.
The investigation takes three days and requires reading git history to reconstruct what was intended. The fix is a migration that requires a business decision about the 3.4%.
What production-grade looks like
Database constraints enforced at the storage layer: CHECK constraints on status columns, foreign key constraints on relationships, NOT NULL on columns that can't be null. State machines modeled explicitly rather than implied by application code. Migration tests that verify the invariants hold after every schema change.
The constraint violation that surfaces as a loud error at write time is infinitely cheaper than the silent corruption that surfaces as a data integrity investigation three months later.
Category 2: Async failure modes
The failure mechanism
Async jobs are the dark matter of production systems. They process in the background, they don't have users waiting on them, and their failure modes are often untested because they're hard to reproduce in development. The visible surface of a system can be working perfectly while a significant fraction of background work is silently failing.
The specific failure modes:
Uncapped retry loops. A job fails transiently, retries, fails again, retries. Without a retry cap or exponential backoff, the retry volume can overwhelm the service that was failing transiently, turning a transient failure into a sustained outage. Without a dead-letter queue, the failed jobs are silently dropped or re-enqueued indefinitely.
Missing idempotency. A job runs to completion, the acknowledgment to the queue is lost, and the job runs again. If the job isn't idempotent — if running it twice produces different results than running it once — the second execution corrupts state. Duplicate emails, double charges, conflicting record updates.
Cascading queue pressure. One type of job fails and re-enqueues. The queue fills with retries of the failed job type, blocking other job types from running. A downstream service that processes payment webhooks stops working because the queue is full of retries from a failed report generation job.
The specific pattern
An analytics pipeline job processes event records in batches of 1,000. The job has a 30-second timeout. An upstream schema change causes a join to become O(n²) at the current data volume. Jobs start timing out at 28 seconds. Each timeout triggers a retry. The retry queue fills. Workers processing other job types — email delivery, webhook dispatch — are blocked behind the analytics retry queue. Email delivery stops. Users with pending email confirmations can't complete signup. The visible failure (email delivery) has no obvious connection to the root cause (analytics query timeout).
The post-mortem takes 6 hours. The fix is two changes: a query optimization and queue priority configuration that should have existed from the start.
What production-grade looks like
Every async job has: explicit retry limits with exponential backoff, a dead-letter queue for exhausted retries, idempotency by design (jobs can be re-run safely), and monitoring on queue depth and job failure rates. Queue priority is configured by job criticality, not first-come-first-served. The dead-letter queue has alerting configured — failed jobs don't disappear silently.
Job failure rates are a first-class metric, not an afterthought. The on-call engineer sees queue health alongside request latency and error rates.
Category 3: External dependency coupling
The failure mechanism
Every external service call is a reliability risk. The application that depends on five external services — a payment processor, an email provider, a storage service, a third-party API, a geolocation service — inherits the failure modes of all five. When any of those services degrades or fails, the failure propagates through the application unless the dependency is properly bounded.
No timeouts. The HTTP client has no configured timeout. A slow external service holds threads open indefinitely. Under load, thread pool exhaustion causes the application to stop accepting new requests.
No circuit breakers. A degraded external service responds slowly but doesn't fail. Without a circuit breaker, every request to that service takes the maximum degraded time rather than failing fast. P99 latency for the application climbs to match the degraded service's response time.
No graceful degradation. When an external service fails, the application fails with it rather than degrading gracefully. The geolocation service is down; every request that uses geolocation returns a 500 rather than omitting the location data.
The specific pattern
A financial reporting platform makes synchronous calls to a third-party data enrichment API during report generation. The API provider has a 2-hour degradation event. Reports that normally generate in 3 seconds take 30 seconds during the event. The report generation endpoint times out on the frontend after 15 seconds. Users get errors trying to generate reports.
The engineering response is to add a timeout to the API call. The timeout surfaces that the enrichment step can fail — and there's no code path for that. The fix for graceful degradation requires refactoring the report generation flow to make enrichment optional.
Both fixes should have been in the original design. The timeout is a 10-minute change. The graceful degradation is a 2-day refactor.
What production-grade looks like
Every external dependency has: a configured timeout (generous enough to accommodate normal latency, strict enough to bound failure impact), circuit breaker logic that fails fast when the service is degraded, and a graceful degradation path that keeps the core application working when the dependency is unavailable.
Dependencies are classified by criticality: blocking (the feature cannot work without it) vs. enriching (the feature degrades but works without it). Enriching dependencies never block the critical path.
Integration tests run against dependency mocks that simulate failure modes, not just happy paths. The slow response, the timeout, the 503 — these are tested before production surfaces them.
Category 4: Observability gaps
The failure mechanism
You can't fix what you can't see. Observability gaps don't cause failures directly — they cause failures to persist longer, cause post-mortems to take longer, and cause the same failures to recur because the root cause was never clearly identified.
The specific gap pattern: logging that doesn't correlate across services, metrics that measure the wrong things, alerting that fires on symptoms rather than causes.
Uncorrelated logs. A request enters the application and spawns three background jobs. Each component logs separately, without a shared request ID. When the failure occurs, reconstructing what happened requires manually correlating timestamps across log files. The investigation that should take 20 minutes takes 3 hours.
Metrics on the wrong things. CPU utilization and memory usage are monitored. Request latency and error rates are not. The first signal of a production problem is a support ticket, not a PagerDuty alert.
Alerting on symptoms. Alerting fires when a specific endpoint starts returning 500s. By the time the 500s are happening, the failure is fully in progress. The alert should have fired when queue depth started climbing, or when database query time exceeded a threshold, or when the error rate on background jobs started rising.
The specific pattern
A multi-tenant platform has an authentication service that starts silently failing for a subset of tenants — those whose organization IDs fall in a specific numeric range due to an integer overflow in a middleware component. The failure affects approximately 8% of tenants.
The platform has CPU and memory monitoring. No request-level metrics. No error rate tracking. The first signal is 15 support tickets in a 2-hour window. The investigation requires correlating support tickets with raw logs across two services. The affected tenant ID range pattern takes 90 minutes to identify.
If request-level metrics had existed, the error rate spike would have been visible in the first 5 minutes. The resolution would have been in 30 minutes, not 4 hours.
What production-grade looks like
Structured logging with request IDs that trace through every component a request touches, including background jobs spawned from that request. Metrics on the things that are operationally meaningful: request latency (P50, P95, P99), error rates by endpoint, queue depths, database query times, background job failure rates.
Alerting on leading indicators, not lagging ones: queue depth climbing, error rate rising, P99 latency increasing — before users notice. Alert thresholds are calibrated so that the alert fires when there's still time to investigate before the failure is fully in progress.
The test: during an incident, can the on-call engineer determine root cause within 15 minutes using only the monitoring tooling? If the answer is no, the observability is insufficient.
The compound failure
The taxonomy exists to be useful — but real production failures are usually compound. The cascade that starts with a single failed job (Category 2) propagates because the failure mode wasn't anticipated in the external dependency (Category 3), and goes undetected for 45 minutes because the monitoring tracks the wrong metrics (Category 4), and the fix takes 3 hours because logs don't correlate (Category 4 again), and during the investigation the team discovers that the data written during the incident is in an invalid state because the constraint enforcement was missing (Category 1).
Each category makes the others worse. They compound.
What this looks like when it's resolved
A system where failures are loud, bounded, and recoverable.
Loud: the on-call engineer knows about a problem before users report it. Metrics and alerting are calibrated to surface failures in their early stage, not after they're fully in progress.
Bounded: a failure in one component doesn't cascade into others. External dependencies fail fast rather than degrading slowly. Queue failures don't starve other queues.
Recoverable: failed jobs have dead-letter queues. Invalid data states are caught at write time, not discovered months later. The runbook for common failure modes exists and is tested.
The specific outcome: mean time to detection under 5 minutes. Mean time to resolution under 30 minutes for known failure modes. Post-mortems that end with the action item "add a constraint" or "add a timeout" — not "redesign the service."
This is for you if
You're a CTO, technical co-founder, or the engineer responsible for a system that's carrying real production load. You've had incidents you'd describe as "came out of nowhere" — which is usually a sign that the failure mode existed but was invisible. You want to understand the taxonomy before the next incident, not during it.
This work is relevant whether you're pre-incident (architecture review against the failure modes above) or post-incident (structured post-mortem and remediation planning). Typical engagement is $100k+ depending on scope and urgency.
This is not for teams that want a surface-level reliability review. It's for teams that want a structural analysis of where the next incident is coming from, and what it takes to prevent it — or at least contain it.
The failures aren't random. They're predictable. The question is whether you predict them or experience them.