问题
Why does the same outage keep recurring?
说出你的本意。只有两位作者都同意,两张卡片才会合并。
已关闭 · 返回行动阶段
正以访客身份查看。输入用户名即可加入并参与。
合并重复卡片
仅限两张提出相同主张的卡片。
为什么
只有两位作者都同意才会合并。合并之前,请先在卡片的侧边讨论里讨论意思。
已合并 "Action items from incident rev…" → "Post-mortem actions are writte…"
Post-mortems assign actions to people who were not in the room
Scope: the services this team runs, not the whole platform.
还没有人发言。
Tests that fail intermittently are retried until green
Customers report outages before monitoring does
I mean the pattern over the last two quarters, not a single incident.
类别 5讨论
The on-call rota has the same two people on it most weeks
类别 1讨论
Leadership asks for a root cause within the hour, so the first plausible one wins
I mean the pattern over the last two quarters, not a single incident.
Reliability has no budget line of its own
To be precise: this is about the process, not the person on call that night.
类别 4讨论
Every team has its own logging format
类别 4讨论
Success is measured in features shipped per quarter
Dependencies are upgraded only when something breaks
Meaning the repeat outages, not the one-off hardware failure in March.
The service map in the wiki is two years old
类别 6讨论
Deploys go out on Friday afternoons because that is when the sprint ends
Post-mortem actions are written up and then never scheduled
类别 2讨论中(1)
Rollbacks take longer than the outage they are meant to end
To be precise: this is about the process, not the person on call that night.
Runbooks are out of date within a month of being written
To be precise: this is about the process, not the person on call that night.
类别 6讨论中(1)
Migrations are run by hand from a laptop
类别 3讨论
Post-mortems assign actions to people who were not in the room
Scope: the services this team runs, not the whole platform.
Feature work always outranks reliability work in planning
Scope: the services this team runs, not the whole platform.
There is no time set aside after an incident, so the write-up is done at midnight
Feature flags are never cleaned up, so nobody knows which paths are live
Meaning the repeat outages, not the one-off hardware failure in March.
类别 2讨论
Config lives in five places and drifts between them
I mean the pattern over the last two quarters, not a single incident.
类别 3讨论
Incidents are declared late because declaring one feels like blame
Load tests were last run before the customer base doubled
The same three services cause most of the pages
类别 1讨论
One engineer knows how the payment path actually works
Scope: the services this team runs, not the whole platform.
Error budgets exist on a slide and nowhere else
The staging environment does not resemble production
The incident channel fills with people asking for status instead of giving it
Nobody owns the shared database schema
To be precise: this is about the process, not the person on call that night.
Retries are unbounded, so a slow dependency becomes a flood
Alerts fire so often that on-call mutes them
I mean the pattern over the last two quarters, not a single incident.
类别 5讨论
Capacity is added after the outage, never before