True Causes
ZH
← 大厅
7/7 · 行动
30 人 · 30 个想法
点击任意已完成的阶段进行查看
✓
提出
✓
澄清
✓
归类
✓
选择
✓
结构
✓
地图
7
行动
演示 · 不计入
障碍 · 持有链接的任何人 · 公开列出 · 已开启联盟
主持人:它的主持人(已封存)
30 人 · 30 个想法 · 2 个小组
通行证 → · 逻辑 → · 记录
↗ 分享这张地图
在其他网站上,作为实时地图:
作为图片: https://truecauses.org/d/6708d57/map.svg
作为数据: 本房间每一个已判断的卡片对(CSV). 每个主张都有自己的地址 /c/6708d57/<card>,列在地图下方。
发布时请注明出处: Causal map: True Causes, https://truecauses.org/d/6708d57
问题

Why does the same outage keep recurring?

提出该做什么,再判断每项行动能否撼动每个根源。

封存于 23 天前:这个房间已不再知道谁曾在其中;只有主持人仍然留名。它得出的结论保留。
30 人在座 (其中 29 名为模拟成员) · 针对 6 个根本原因已提出 3 项行动。添加一项,然后判断每项行动能否撼动每个原因。
正以访客身份查看。输入用户名即可加入并参与。
先给根本原因打分
为什么
本方法自身的做法,在提出任何行动之前:每个根本原因按 1–5 分打分——解决它的可行性有多高、解决它会带来多大影响、不加干预它自行解决的可能性有多大。优先考虑影响大、可行性高、自行解决可能性低的根源——优先级 = 影响 + 可行性 − 可能性。分数在小组内取平均,并显示有多少人打了分。
1 = 低,5 = 高。这是你自己的分数;旁边是小组的平均分。
根本原因可行性影响自行解决优先级
The service map in the wiki is two years old––––
One engineer knows how the payment path actually works––––
Feature flags are never cleaned up, so nobody knows which paths are live––––
The on-call rota has the same two people on it most weeks––––
Rollbacks take longer than the outage they are meant to end––––
Alerts fire so often that on-call mutes them––––
每项行动能撼动每个根本原因吗?
竖列:大家提出的行动。横行:地图找出的根本原因。每个格子都是对一个问题的一次投票——这项行动能否显著减少那个原因?一方必须达到所投票数的 75% 才算定下;达不到,格子就保持分裂。 每个格子显示其投票分布——是 / 更糟 / 否——当其中一项达到四分之三时,该格子就定下了。
为什么
以所投票数的 75% 定下,和这里的其他所有问题一样。分裂表示有人投了票,但没有一方达到这个标准——这是一个结果,不是排队等待,它会一直保持分裂,直到足够多的人改变主意。没有人定下的格子保持未定;它永远不会变成否。逐对单独判断,能防止一项受欢迎的行动被记上它根本没触及的功劳。
行动根本原因 · 左边的行动能减少它吗?
The service map in the wiki is two years oldOne engineer knows how the payment path actually worksFeature flags are never cleaned up, so nobody knows which paths are liveThe on-call rota has the same two people on it most weeksRollbacks take longer than the outage they are meant to endAlerts fire so often that on-call mutes them
No production deploy after 14:00 on Fridays unless the on-call engineer agrees 讨论✗ 不,不能
3 是 / 0 更糟 / 26 否
⚑ 小组意见分裂
21 是 / 0 更糟 / 8 否
尚未确定
尚未确定
尚未确定
尚未确定
Give the shared schema one named owner and a migration tool with rollback 讨论✗ 不,不能
6 是 / 0 更糟 / 23 否
✓ 是,它能减少这一项
24 是 / 0 更糟 / 5 否
尚未确定
尚未确定
尚未确定
尚未确定
Reserve one engineer-week per sprint for post-mortem actions, first in the sprint, not last 讨论✗ 不,不能
4 是 / 0 更糟 / 25 否
✗ 不,不能
3 是 / 0 更糟 / 26 否
尚未确定
尚未确定
尚未确定
尚未确定
目前杠杆作用最大
Give the shared schema one named owner and a migration tool with rollback 减少了 6 个根本原因中的 1 个
地图上的行动
显示的是房间判断过的所有行动:它直接作用于 6 个根本原因中的 1 个。 有 5 个没有直接作用。
只依据房间在行动阶段定下的格子,别无其他。绿色的根本原因至少被所显示行动中的一项减轻;灰色的,没有一项直接作用于它;红色边框表示其中某项会使它更糟。
The same three services cause most of the pages → The staging environment does not resemble productionFeature work always outranks reliability work in planning → Post-mortem actions are written up and then never scheduledRollbacks take longer than the outage they are meant to end → Reliability has no budget line of its ownEvery team has its own logging format → Customers report outages before monitoring doesReliability has no budget line of its own → The incident channel fills with people asking for status instead of giving itReliability has no budget line of its own → The staging environment does not resemble productionPost-mortems assign actions to people who were not in the room → Post-mortem actions are written up and then never scheduledTests that fail intermittently are retried until green → Customers report outages before monitoring doesRetries are unbounded, so a slow dependency becomes a flood → The same three services cause most of the pagesAlerts fire so often that on-call mutes them → Every team has its own logging formatThe on-call rota has the same two people on it most weeks → Post-mortem actions are written up and then never scheduledThe on-call rota has the same two people on it most weeks → Alerts fire so often that on-call mutes themFeature flags are never cleaned up, so nobody knows which paths are live → The same three services cause most of the pagesFeature flags are never cleaned up, so nobody knows which paths are live → Rollbacks take longer than the outage they are meant to endOne engineer knows how the payment path actually works → The on-call rota has the same two people on it most weeksThe service map in the wiki is two years old → Feature flags are never cleaned up, so nobody knows which paths are liveFeature work always outranks reliability work in planningFeature work alwaysoutranks reliability workin planningPost-mortems assign actions to people who were not in the roomPost-mortems assign actionsto people who were not inthe roomTests that fail intermittently are retried until greenTests that failintermittently are retrieduntil greenRetries are unbounded, so a slow dependency becomes a floodRetries are unbounded, so aslow dependency becomes afloodOne engineer knows how the payment path actually works ② Give the shared schema one named owner and a migration tool with rollbackOne engineer knows how thepayment path actually works②所显示的行动对这个根本原因的作用1 项行动减轻它The service map in the wiki is two years oldThe service map in the wikiis two years old所显示的行动对这个根本原因的作用没有行动直接作用于它Feature flags are never cleaned up, so nobody knows which paths are liveFeature flags are nevercleaned up, so nobody knowswhich paths are live所显示的行动对这个根本原因的作用没有行动直接作用于它The on-call rota has the same two people on it most weeksThe on-call rota has thesame two people on it mostweeks所显示的行动对这个根本原因的作用没有行动直接作用于它Rollbacks take longer than the outage they are meant to endRollbacks take longer thanthe outage they are meantto end所显示的行动对这个根本原因的作用没有行动直接作用于它Alerts fire so often that on-call mutes themAlerts fire so often thaton-call mutes them所显示的行动对这个根本原因的作用没有行动直接作用于它The same three services cause most of the pagesThe same three servicescause most of the pagesPost-mortem actions are written up and then never scheduledPost-mortem actions arewritten up and then neverscheduledReliability has no budget line of its ownReliability has no budgetline of its ownEvery team has its own logging formatEvery team has its ownlogging formatThe incident channel fills with people asking for status instead of giving itThe incident channel fillswith people asking forstatus instead of giving itThe staging environment does not resemble productionThe staging environmentdoes not resembleproductionCustomers report outages before monitoring doesCustomers report outagesbefore monitoring does
地图上的每个编号标出该行动所减轻的一个根本原因。
① No production deploy after 14:00 on Fridays unless the on-call engineer agrees
② Give the shared schema one named owner and a migration tool with rollback
③ Reserve one engineer-week per sprint for post-mortem actions, first in the sprint, not last
我们预期接下来会发生什么
关于各项行动的预测:每项应改变什么、如何衡量、何时检查。到了那一天,曾在房间里的某个人会在旁边写下实际发生了什么。
为什么
地图说明哪些行动能撼动哪些原因。它不说明是否有人做了这些行动,也不说明是否有什么改变了。这里的每一条都是一项预测,带有日期和要衡量的东西,在地图定下后写成。日期到了,实际发生的情况会写在旁边,而预测保持原样:错误的预测才是有用的那种。没有人对这些投过票,它也不改变房间发现的任何东西。
到期:23 Jan 2027
行动No production deploy after 14:00 on Fridays unless the on-call engineer agrees
预期the same cause stops coming back once fixing it is in the sprint
衡量指标repeat outages with the same cause, per quarter
估计房间:3(15 人的中位数,范围 1 到 6)