Self-healing multitier architectures using cascading rescue points

Abstract

Software bugs and vulnerabilities cause serious problems to both home users and the Internet infrastructure, limiting the availability of Internet services, causing loss of data, and reducing system integrity. Software self-healing using rescue points (RPs) is a known mechanism for recovering from unforeseen errors. However, applying it on multitier architectures can be problematic because certain actions, like transmitting data over the network, cannot be undone. We propose cascading rescue points (CRPs) to address the state inconsistency issues that can arise when using traditional RPs to recover from errors in interconnected applications. With CRPs, when an application executing within a RP transmits data, the remote peer is notified to also perform a checkpoint, so the communicating entities checkpoint in a coordinated, but loosely coupled way. Notifications are also sent when RPs successfully complete execution, and when recovery is initiated, so that the appropriate action is performed by remote parties. We developed a tool that implements CRPs by dynamically instrumenting binaries and transparently injecting notifications in the already established TCP channels between applications. We tested our tool with various applications, including the MySQL and Apache servers, and show that it allows them to successfully recover from errors, while incurring moderate overhead between 4.54% and 71.56%.

Speculative Memory Checkpointing

“…We evaluated our SMC framework implementation on a workstation running Linux v3.12.36 (x64) and equipped with a dual-core Intel Pentium G6950 2.80 GHz processor and 16 GB RAM. To evaluate the real-world impact of SMC, we selected five popular server programs-a common target for memory checkpointing applications in prior work in the area [20,53,54,58,70]-and allowed our deployed SMC framework to checkpoint the memory image of their worker processes at every client request, following the common request-oriented checkpointing model [22,39]. For our analysis, we considered the three most popular open-source web servers-Apache httpd (v.2.2.23), nginx (v0.8.54), and lighttpd (v1.4.28)-a popular RDBMS server-PostgreSQL (v9.0.10)-and a widely used DNS server-BIND (v9.9.3).…”

“…The data (and metadata) associated with every checkpoint is maintained in an in-kernel journal by the core checkpointing component (CKPT ) of ksmc. The journal stores a maximum predetermined number of K checkpoints on a per-process basis, following a FIFO replacement strategy-currently K = 1 by default, a common assumption in traditional memory checkpointing applications [20,22,32,39,53,54,58,68,70]. When necessary, user programs can issue a memory restore request and allow ksmc to automatically revert the current memory image to the last checkpoint k, with k ∈ [1; K].…”

“…Memory checkpointing-the ability to snapshot/restore the memory image of a running process or set of processes-has recently gained momentum in several application domains. In automatic error recovery applications, memory checkpointing enables fast and safe recovery to known and stable program states [20,22,23,32,39,53,54,57,58,62,70]. In debugging applications, it enables users to efficiently navigate through several program states observed during the execution, while empowering advanced debugging techniques such as reverse/replay debugging [27,34,60,61].…”

“…Traditional memory checkpointing techniques rely on commodity hardware-a strategy that provides superior deployability compared to instrumentation-based strategies [10,14,39,40,53,64,65,70,76]-to incrementally copy memory pages that were modified by the running program [11,20,21,34,37,51,54,56,58,60,63]. While incremental memory checkpointing is regarded as an efficient alternative to disk-based or full memory checkpointing [52], it still incurs nontrivial memory tracing costs for every taken checkpoint, resulting in relatively infrequent checkpoints used in practice.…”

“…Traditional incremental checkpointing mechanisms are usually page-granular, that is, a memory page is the smallest data block copied (although more fine-grained techniques exist [17,39,53,70,76]). Below we discuss the core mechanisms and techniques employed by these approaches.…”