Weathering the unexpected

Krishnan, K. Sarojani

doi:10.1145/2366316.2366332

Cited by 5 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use regression testing before rolling out software updates and deploy canaries at smaller scales before deploying to the entire network. We also periodically exercise disaster scenarios [20,17] and enhance our systems based on lessons from these exercises. We carefully document every management operation (MOp) on the network.…”

Section: Baseline Availability Mechanismsmentioning

confidence: 99%

Evolve or Die

Govindan

Minei

Kallahalla

et al. 2016

Proceedings of the 2016 ACM SIGCOMM Conference

167

View full text Add to dashboard Cite

Maintaining the highest levels of availability for content providers is challenging in the face of scale, network evolution, and complexity. Little, however, is known about the network failures large content providers are susceptible to, and what mechanisms they employ to ensure high availability. From a detailed analysis of over 100 high-impact failure events within Google's network, encompassing many data centers and two WANs, we quantify several dimensions of availability failures. We find that failures are evenly distributed across different network types and across data, control, and management planes, but that a large number of failures happen when a network management operation is in progress within the network. We discuss some of these failures in detail, and also describe our design principles for high availability motivated by these failures. These include using defense in depth, maintaining consistency across planes, failing open on large failures, carefully preventing and avoiding failures, and assessing root cause quickly. Our findings suggest that, as networks become more complicated, failures lurk everywhere, and, counter-intuitively, continuous incremental evolution of the network can, when applied together with our design principles, result in a more robust network.

show abstract

Section: Baseline Availability Mechanismsmentioning

confidence: 99%

Evolve or Die

Govindan

Minei

Kallahalla

et al. 2016

Proceedings of the 2016 ACM SIGCOMM Conference

167

View full text Add to dashboard Cite

show abstract

“…Different from public WANs, backbone link capacity at Facebook is not a constraint (similar to other private backbone networks [18-20, 22, 24, 29, 44]), especially given that the traffic induced by dynamic content is only a small percentage of the overall bandwidth. Note that backbone capacity is constantly verified by regular load testing [53] and drain/DiRT-like testing [30,54] that manipulate live traffic at edge nodes to simulate how data centers handle worst-case scenarios. Compared with the amount of traffic driven by these tests, the amount of dynamic user traffic Taiji manages does not stress our backbone links.…”

Section: Introductionmentioning

confidence: 99%

Taiji

Chou

Veeraraghavan

et al. 2019

Proceedings of the 27th ACM Symposium on Operating Systems Principles

View full text Add to dashboard Cite

We present Taiji, a new system for managing user traffic for large-scale Internet services that accomplishes two goals: 1) balancing the utilization of data centers and 2) minimizing network latency of user requests. Taiji models edge-to-datacenter traffic routing as an assignment problem-assigning traffic objects at the edge to the data centers to satisfy service-level objectives. Taiji uses a constraint optimization solver to generate an optimal routing table that specifies the fractions of traffic each edge node will distribute to different data centers. Taiji continuously adjusts the routing table to accommodate the dynamics of user traffic and failure events that reduce capacity. Taiji leverages connections among users to selectively route traffic of highly-connected users to the same data centers based on fractions in the routing table. This routing strategy, which we term connection-aware routing, allows us to reduce query load on our backend storage by 17%. Taiji has been used in production at Facebook for more than four years and routes global traffic in a user-aware manner for several large-scale product services across dozens of edge nodes and data centers.

show abstract

“…Tolerance to any kind of service disruption, whether caused by a simple hardware fault or by a largescale disaster, is key for the survival of modern distributed systems. Cloud-scale applications must be inherently resilient, as any outage has direct implications on the business behind them [24].…”

Section: Introductionmentioning

confidence: 99%

XFT: Practical Fault Tolerance Beyond Crashes

Liu¹,

Viotti²,

Cachin³

et al. 2015

Preprint

View full text Add to dashboard Cite

Despite years of intensive research, Byzantine fault-tolerant (BFT) systems have not yet been adopted in practice. This is due to additional cost of BFT in terms of resources, protocol complexity and performance, compared with crash fault-tolerance (CFT). This overhead of BFT comes from the assumption of a powerful adversary that can fully control not only the Byzantine faulty machines, but at the same time also the message delivery schedule across the entire network, effectively inducing communication asynchrony and partitioning otherwise correct machines at will. To many practitioners, however, such strong attacks appear irrelevant.In this paper, we introduce cross fault tolerance or XFT, a novel approach to building reliable and secure distributed systems and apply it to the classical state-machine replication (SMR) problem. In short, an XFT SMR protocol provides the reliability guarantees of widely used asynchronous CFT SMR protocols such as Paxos and Raft, but also tolerates Byzantine faults in combination with network asynchrony, as long as a majority of replicas are correct and communicate synchronously. This allows the development of XFT systems at the price of CFT (already paid for in practice), yet with strictly stronger resilience than CFT -sometimes even stronger than BFT itself.As a showcase for XFT, we present XPaxos, the first XFT SMR protocol, and deploy it in a geo-replicated setting. Although it offers much stronger resilience than CFT SMR at no extra resource cost, the performance of XPaxos matches that of the state-of-the-art CFT protocols. * Work done while being a PhD student at EURECOM. 1 As an illustration, five nines reliability means that a system is up and correctly running at least 99.999% of the time. In other words, malfunction is limited to one hour every 10 years on average.

show abstract

Weathering the unexpected

Abstract: Failures happen, and resilience drills help organizations prepare for them.

Cited by 5 publications

References 0 publications

Evolve or Die

Evolve or Die

Taiji

XFT: Practical Fault Tolerance Beyond Crashes

Contact Info

Product

Resources

About