Transparently consistent asynchronous shared memory

Abstract

The advent of many-core processors is imposing many changes on the operating system. The resources that are under contention have changed; previously, CPU cycles were the resource in demand and required fair and precise sharing. Now compute cycles are plentiful, but the memory per core is decreasing. In the past, scientific applications used all the CPU cores to finish as fast as possible, with visualization and analysis of the data performed after the simulation finished. With decreasing memory available per core, as well as the higher price (in power and time) for storing data on disk or sending it over the network, it now makes sense to run visualization and analytics applications in-situ, while the application is running. Visualization and analytics applications then need to sample the simulation memory with as little interference and as little changes in the simulation code as possible.We propose an asynchronous memory sharing facility that allows consistent states of the memory to be shared between processes without any implicit or explicit synchronization. We distinguish two types of processes; a single producer and one or more observers. The producer modifies the state of the data, making available consistent versions of the state to any observer. The observers, working at different sampling rates, can access the latest available consistent state.Some applications that would benefit from this type of facility include check-pointing applications, processes monitoring, unobtrusive process debugging, and the sharing of data for visualization or analytics. To evaluate our ideas we have developed two kernel-level implementations for sharing data asynchronously and we compared these implementations to a traditional user-space synchronized multi-buffer method.We have seen improvements of up to 3.5x in our tests over the traditional multi-buffer method with 20% of the data pages touched.

A Cross-Enclave Composition Mechanism for Exascale System Software

“…TCASM Transparently Consistent Asynchronous Shared Memory (TCASM) [1] is a shared memory based transport that allows a producer process, typically an HPC application, to share read-only data in situ with an observer process, typically analytics or checkpointing, without needing to rely on manual synchronization between the producing and observing process. TCASM is implemented using copy on write semantics: the producer registers a region of memory corresponding to its data and then exports a snapshot of that memory when it reaches a consistent state.…”

“…It creates no-op stub functions (e.g., functions which do no work and return) for the composition API in our composed applications to find a lower bound for performance. We then try a composed COW approach to composition on Linux using TCASM [1]. Finally we test measure against a "worst case" that uses spinlocks with no backo↵ to serialize the the computation and data exchange in SNAP and its analytics over an XPMEM [20].…”

System-Level Support for Composition of Applications

“…Our approach provides the runtime provisioning of isolated enclave instances that can be customized to support each application component. Additionally, our approach allows application composition through the use of cross enclave shared memory segments through the XEMEM shared memory system [20], whose application interface can be accessed using existing I/O mechanisms such as ADIOS [24] or TCASM [5].…”

“…Memory (TCASM) [5] is a shared memory based transport that allows a producer process, typically an HPC application, to share read-only data in situ with an observer process, typically analytics or checkpointing, without needing to rely on manual synchronization between the producing and observing process. TCASM is implemented using copy on write semantics: the producer registers a region of memory corresponding to its data and then exports a snapshot of that memory when it reaches a consistent state.…”

What is a Lightweight Kernel?

“…There are several projects that look at combining Linux functionality with an LWK. Kitten/Palacios, Fused OS, McKernel, and mOS are driven by anticipated usage models of future machines [20,4]. The projects differ in how they provide Linux functionality while achieving LWK performance and scalability.…”