Unraveling Network-Induced Memory Contention: Deeper Insights with Machine Learning

Abstract

Abstract-Remote Direct Memory Access (RDMA) is expected to be an integral communication mechanism for future exascale systems -enabling asynchronous data transfers, so that applications may fully utilize CPU resources while simultaneously sharing data amongst remote nodes. In this work we examine Network-induced Memory Contention (NiMC) on Infiniband networks. We expose the interactions between RDMA, main-memory and cache, when applications and out-of-band services compete for memory resources. We then explore NiMC's resulting impact on application-level performance. For a range of hardware technologies and HPC workloads, we quantify NiMC and show that NiMC's impact grows with scale resulting in up to 3X performance degradation at scales as small as 8K processes even in applications that previously have been shown to be performance resilient in the presence of noise. Additionally, this work examines the problem of predicting NiMC's impact on applications by leveraging machine learning and easily accessible performance counters. This approach provides additional insights about the root cause of NiMC and facilitates dynamic selection of potential solutions. Lastly, we evaluated three potential techniques to reduce NiMC's impact, namely hardware offloading, core reservation and software-based network throttling.

Anomaly Detection and Anticipation in High Performance Computing Systems

“…Preliminary research works addressing anomaly detection and fault prediction using real anomalies have been proposed, but they tend to focus on the availability of a single (HW or SW) component of the system and not on the availability of the entire computing nodes. Ostruckow et al [32] analyze the (specific) failures of GPU processors, Boixaderas et al [23] aim at predicting the memory (DRAM) failures, Di et al [33] detect silent data corruption, Groves et al [34] predict sub-optimal operation due to memory contention. These approaches focus on specific HW components, but in the age of Exascale, with larger systems, more components and higher costs, such partial detection and monitoring systems should be combined and enhanced with supervised and unsupervised holistic anomaly detection models [14], [24].…”

Use It or Lose It: Cheap Compute Everywhere

“…In work by Phothilimthana et al [52], a DPU was used as a write-back cache for a host-based key-value store (KVS) and observed a 28-60% performance improvement. By offloading to portions of a KVS, host applications may experience less pollution of the cache due to network-induced memory contention [25]. Despite the benefits, a challenge of utilizing the DPU as a cache is maintaining a consistent state between the data that resides on the DPU and host memory.…”