Failing and Learning: A Study of What is Learned About Reliability From Software Incidents

Many organizations have defined processes for learning from failure, often involving postmortem analyses conducted after any system failures that are judged to be sufficiently severe. The output from such a postmortem is an incident report that describes in detail the failure and what was learned about the failure through the postmortem process. To better understand what is learned about the reliability of systems from failures, we have conducted a qualitative analysis of 89 failures (documented in 70 incident reports), with a particular focus on what is learned about the role of people, such as designers, operators and incident responders. Based on our analysis, this paper reports on insights related to three reliability topics: (1) the events that trigger incidents, (2) the contributing factors (sometimes called root causes) of incidents, and (3) the recovery from failures. Our results provide insights into the contributions people make toward system reliability, the challenges they face, and also how a failure-driven learning process informs people’s work.