Fault Injection in Production

Abstract

When we build Web infrastructures at Etsy, we aim to make them resilient. This means designing them carefully so that they can sustain their (increasingly critical) operations in the face of failure.Thankfully, there have been a couple of decades and reams of paper spent on researching how fault tolerance and graceful degradation can be brought to computer systems. That helps the cause.To make sure that the resilience built into Etsy systems is sound and that the systems behave as expected, we have to see the failures being tolerated in production.Why production? Why not simulate this in a QA or staging environment? First, the existence of any differences in those environments brings uncertainty to the exercise, and second, the risk of not recovering has no consequences during testing, which can bring hidden assumptions into the faulttolerance design and into recovery. The goal is to reduce uncertainty, not increase it.Forcing failures to happen, or even designing systems to fail on their own, generally isn't easily sold to management. Engineers are not conditioned to embrace their ability to respond to emergencies; they aim to avoid them altogether. Taking a detailed look at how to respond better to failure is essentially accepting that failure will happen, which you might think is counter to what you want in engineering, or in business.Take, for example, what you would normally think of as a simple case: the provisioning of a server or cloud instance from zero to production:1. Bare metal (or cloud-compute instance) is made available.2. Base operating system is installed via PXE (preboot execution environment) or machine image.3. Operating-system-level configurations are put into place (via configuration management or machine image).4. Application-level configurations are put into place (via configuration management, app deployment, or machine image). 5. Application code is put into place and underlying services are started correctly (via configuration management, app deployment, or machine image). 6. Systems integration takes place in the network (load balancers, VLANs, routing, switching, DNS, etc.). This is probably an oversimplification, and each step or layer is likely to represent a multitude of CPU cycles; disk, network and/or memory operations; and various numbers of software mechanisms.All of these come together to bring a node into production.Operability means that you can have confidence in this node coming into production, possibly joining a cluster, and serving live traffic seamlessly every time it happens. Furthermore, you want and expect to have confidence that if the underlying power, configuration, application, or compute resources (CPU, disk, memory, network, etc.) experience a fault, then you can survive such a fault by some means: allowing the application to degrade gracefully, rebuild itself, take itself out of production, and alert on the specifics of the fault, etc.At a high level, production fault injection should be considered one of many approaches used to gain confidence in the safety ...

How Do Organizations Learn from Information System Incidents? A Synthesis of the Past, Present, and Future

“…Embedded learning practices include attending mindfully to the incident in its local setting, developing emergent strategies to address unpredictable situations, and gaining firsthand experience which, over time, expands the organization's repertoire of practices for handling future incidents. An interesting learning practice in this mode is suggested by Allspaw (2012), who advocated that organizations intentionally inject faults into their systems to engage their members in handling incidents, which can enhance their practical capabilities. Box 3 further illustrates this learning mode.…”

Principles of Antifragile Software

“…They call this practice "chaos engineering". When fault injection is done in production on a special day under full control (as opposed to automatically at any arbitrary point in time), it is called a GameDay exercise [1].…”

The Risks of Autonomy

“…Handling life cycle dynamics will require an architecture equipped with the capacity to adjust performance over a wide dynamic range (Doyle & Csete, 2011). This is, in part, the target of extensible critical digital services (Allspaw, 2012) and closely related to resilience engineering (Woods, 2015). Closing the gap between the demonstration and the real thing requires the development of new ways to design systems to be manageable and extensible over life cycles.…”