Why does the same outage keep recurring?
a walkthrough with stand-ins — a demonstration, not a finding · 20 certified links · outlined boxes are the roots · tap a claim to read it in full
Showing every action the room judged: it acts directly on 1 of the 6 root causes. It does not act directly on 5.
The same three services cause most of the pages → The staging environment does not resemble productionFeature work always outranks reliability work in planning → Post-mortem actions are written up and then never scheduledRollbacks take longer than the outage they are meant to end → Reliability has no budget line of its ownEvery team has its own logging format → Customers report outages before monitoring doesReliability has no budget line of its own → The incident channel fills with people asking for status instead of giving itReliability has no budget line of its own → The staging environment does not resemble productionPost-mortems assign actions to people who were not in the room → Post-mortem actions are written up and then never scheduledTests that fail intermittently are retried until green → Customers report outages before monitoring doesRetries are unbounded, so a slow dependency becomes a flood → The same three services cause most of the pagesAlerts fire so often that on-call mutes them → Every team has its own logging formatThe on-call rota has the same two people on it most weeks → Post-mortem actions are written up and then never scheduledThe on-call rota has the same two people on it most weeks → Alerts fire so often that on-call mutes themFeature flags are never cleaned up, so nobody knows which paths are live → The same three services cause most of the pagesFeature flags are never cleaned up, so nobody knows which paths are live → Rollbacks take longer than the outage they are meant to endOne engineer knows how the payment path actually works → The on-call rota has the same two people on it most weeksThe service map in the wiki is two years old → Feature flags are never cleaned up, so nobody knows which paths are liveFeature work always outranks reliability work in planningFeature work alwaysoutranks reliability workin planningPost-mortems assign actions to people who were not in the roomPost-mortems assign actionsto people who were not inthe roomTests that fail intermittently are retried until greenTests that failintermittently are retrieduntil greenRetries are unbounded, so a slow dependency becomes a floodRetries are unbounded, so aslow dependency becomes afloodOne engineer knows how the payment path actually worksOne engineer knows how thepayment path actually works②what the shown actions do to this root causeREDUCED BY 1 ACTIONThe service map in the wiki is two years oldThe service map in the wikiis two years oldwhat the shown actions do to this root causeNO ACTION ACTS ON ITDIRECTLYFeature flags are never cleaned up, so nobody knows which paths are liveFeature flags are nevercleaned up, so nobody knowswhich paths are livewhat the shown actions do to this root causeNO ACTION ACTS ON ITDIRECTLYThe on-call rota has the same two people on it most weeksThe on-call rota has thesame two people on it mostweekswhat the shown actions do to this root causeNO ACTION ACTS ON ITDIRECTLYRollbacks take longer than the outage they are meant to endRollbacks take longer thanthe outage they are meantto endwhat the shown actions do to this root causeNO ACTION ACTS ON ITDIRECTLYAlerts fire so often that on-call mutes themAlerts fire so often thaton-call mutes themwhat the shown actions do to this root causeNO ACTION ACTS ON ITDIRECTLYThe same three services cause most of the pagesThe same three servicescause most of the pagesPost-mortem actions are written up and then never scheduledPost-mortem actions arewritten up and then neverscheduledReliability has no budget line of its ownReliability has no budgetline of its ownEvery team has its own logging formatEvery team has its ownlogging formatThe incident channel fills with people asking for status instead of giving itThe incident channel fillswith people asking forstatus instead of giving itThe staging environment does not resemble productionThe staging environmentdoes not resembleproductionCustomers report outages before monitoring doesCustomers report outagesbefore monitoring does
WHAT WAS PREDICTED TO FOLLOW
Each number on the map marks a root cause that action reduces.
① No production deploy after 14:00 on Fridays unless the on-call engineer agrees
we expect: the same cause stops coming back once fixing it is in the sprint
measured by: repeat outages with the same cause, per quarter
due 23 Jan 2027
guesses: the room: 3 (median of 15, spread 1 to 6)
② Give the shared schema one named owner and a migration tool with rollback
no prediction about this action yet
③ Reserve one engineer-week per sprint for post-mortem actions, first in the sprint, not last
no prediction about this action yet
A map from True Causes: a group’s judgement, not a measurement.