C-MemMAP: clustering-driven compact, adaptable, and generalizable meta-LSTM models for memory access prediction

Abstract

With the rise of Big Data, there has been a significant effort in increasing compute power through GPUs, TPUs, and heterogeneous architectures. As a result, many applications are memory bound, i.e., they are bottlenecked by the movement of data from main memory to compute units. One way to address this issue is through data prefetching, which relies on accurate prediction of memory accesses. While recent deep learning models have performed well on sequence prediction problems, they are far too heavy in terms of model size and inference latency to be practical for data prefetching. Here, we propose clustering-driven compact LSTM models that can predict the next memory access with high accuracy. We introduce a novel clustering approach called Delegated model that can reliably cluster the applications. For each cluster, we train a compact meta-LSTM model that can quickly adapt to any application in the cluster. Prior LSTM-based work on access prediction has used orders of magnitude more parameters and developed one model for each application (trace). While one (specialized) model per application can result in more accuracy, it is not a scalable approach. In contrast, our models can predict for a class of applications by trading off specialization at the cost of few retraining steps at runtime, for a more generalizable compact meta-model. Our experiments on 13 benchmark applications demonstrate that clustering-driven ensemble compact meta-models can obtain accuracy close to specialized models using few batches of retraining for majority of the applications.

Fine-grained address segmentation for attention-based variable-degree prefetching

“…RAOP [74] leverages LSTM-based models for virtual address predictions. C-MemMAP [75] combines clustering and meta-models to reduce the model size. Seq2seq modeling [40] based on LSTM encoder-decoder structure has been applied for memory sequence prediction.…”

“…Shi et al [56] propose Voyager that predicts both page sequence and page offsets using two LSTM models along with a dot-product attention mechanism. Existing ML-based prefetchers use history memory access sequence to predict the next memory access address [3,16,46,56,61,62,[73][74][75], which leads to a prefetch degree as one. These models require recurrent greedy/beam search or accepting low-probability candidates to realize higher degree prefetching.…”

“…Machine learning-based data prefetchers are gaining increasing attention to pursue higher performance for memory access prediction [15,16,61] and prefetching [3,45,46,56,61,73]. Prefetching is commonly modeled as classification for sequence prediction [3,15,16,46,56,61,74,75], which is analogous to the problem setting of text prediction [15] in natural language processing (NLP) [39]. However, this analogy is not perfect.…”

“…Second, tokenization [68], as a commonly used technique in NLP that maps a meaningful word into nonsensitive numerical data for model processing, is also borrowed by existing ML-based prefetching models for preprocessing. Tokenization results in extra storage to save the mapping tables (token dictionaries) in hardware implementation, but the cost has been neglected by previous works [3,15,16,46,56,61,74,75]. Third, unlike text prediction with a ground truth of future words in a fixed order, in prefetching there is no ground truth of a certain memory address that should be prefetched.…”

“…Lastly, the latency overhead of ML-based prefetcher is also ignored under this setting. LSTM (Long short-term memory) [18] is a commonly used prediction model [16,56,61,62,74,75] due to its advantage in sequence modeling. However, the recurrent structure of LSTM is hard to be parallelized and the inference latency increases linearly with the input time step length.…”