Cores that don't count

Abstract

We are accustomed to thinking of computers as fail-stop, especially the cores that execute instructions, and most system software implicitly relies on that assumption. During most of the VLSI era, processors that passed manufacturing tests and were operated within specifications have insulated us from this fiction. As fabrication pushes towards smaller feature sizes and more elaborate computational structures, and as increasingly specialized instruction-silicon pairings are introduced to improve performance, we have observed ephemeral computational errors that were not detected during manufacturing tests. These defects cannot always be mitigated by techniques such as microcode updates, and may be correlated to specific components within the processor, allowing small code changes to effect large shifts in reliability. Worse, these failures are often "silent" -the only symptom is an erroneous computation.We refer to a core that develops such behavior as "mercurial." Mercurial cores are extremely rare, but in a large fleet of servers we can observe the disruption they cause, often enough to see them as a distinct problem -one that will require collaboration between hardware designers, processor vendors, and systems software architects.This paper is a call-to-action for a new focus in systems research; we speculate about several software-based approaches to mercurial cores, ranging from better detection and isolating mechanisms, to methods for tolerating the silent data corruption they cause.

New Horizons for High-Performance Computing

“…This will undoubtedly create new resilience issues in future systems. 4,5 For academia and/or researchers, the reaction is consistent with Hennessy and Patterson's description of a renaissance in computer architectures. 6 New horizons to explore!…”

Availability analysis of redundant and replicated cloud services with Bayesian networks

“…First, many vendors already provide mean time to failure (MTTF) information for their software or hardware components. Secondly, cloud providers host larger numbers of hardware components in their data centers, which are constantly monitored, providing significant amounts of data also for rare events [34]. Thirdly, for yet unobserved failures of highly available components, one can use rare event analysis (an active research area) in conjunction with expert knowledge acquisition to incorporate prior beliefs first and later refine the estimate with observation during mission time.…”

Reinventing High Performance Computing: Challenges and Opportunities

“…As cloud scale, complexity, and operational experience continued to grow, additional optimization and leverage opportunities emerged, including software defined networking [46], protocol offloads, and custom network architectures (greatly reducing dependence on traditional network hardware vendors) [47]; quantitative analysis of processor [48], memory [49,50], network [51,52] and disk failure modes [53,54], with consequent redesign for reliability and lower cost (dictating specifications to vendors via consortia like Open Compute [55]); custom processor SKUs, custom accelerators (FPGAs and ASICs), and finally, complete processor design (e.g., Apple silicon, Google TPUs [56] and AWS Gravitons). In between, the cloud vendors deployed their own global fiber networks.…”