Cross-Stack Workload Characterization of Deep Recommendation Systems

Hsia, Samuel; Gupta, Udit; Wilkening, Mark; Wu, Carole-Jean; Wei, Gu-Yeon; Brooks, David

doi:10.1109/iiswc50251.2020.00024

Cited by 22 publications

(13 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For commensurate analysis, final results are presented based on the highest quality achieved for each model and dataset: NDCG of 92. 25 for Criteo (see Section 2). • Tail-latency: To maintain user-experience, recommendations must meet SLAs and be served under strict taillatency targets [16], measured as 99 th percentile (p99).…”

Section: Experimental Methodologymentioning

confidence: 99%

“…Lots of research effort has been devoted to design specialized hardware for deep learning-especially MLPs, CNNs, and RNNs [4,6,7,8,17,19,30,42,43,44,47,54,55]. However, recommendation systems pose distinct challenges owing to their network architectures and use cases [18,25,41]. Given its importance, hardware proposals for accelerating recommendation models have begun to emerge [1,3,9,10,16,26,29,33,34,36,37,40,50].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance

Gupta¹,

Hsia²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning recommendation systems must provide high quality, personalized content under strict tail-latency targets and high system loads. This paper presents RecPipe, a system to jointly optimize recommendation quality and inference performance. Central to RecPipe is decomposing recommendation models into multi-stage pipelines to maintain quality while reducing compute complexity and exposing distinct parallelism opportunities. RecPipe implements an inference scheduler to map multi-stage recommendation engines onto commodity, heterogeneous platforms (e.g., CPUs, GPUs). While the hardware-aware scheduling improves ranking efficiency, the commodity platforms suffer from many limitations requiring specialized hardware. Thus, we design RecPipeAccel (RPAccel), a custom accelerator that jointly optimizes quality, tail-latency, and system throughput. RPAccel is designed specifically to exploit the distinct design space opened via RecPipe. In particular, RPAccel processes queries in sub-batches to pipeline recommendation stages, implements dual static and dynamic embedding caches, a set of top-k filtering units, and a reconfigurable systolic array. Compared to prior-art and at iso-quality, we demonstrate that RPAccel improves latency and throughput by 3× and 6×.

show abstract

Section: Experimental Methodologymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance

Gupta¹,

Hsia²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It has also been adopted by the industry-wide MLPerf benchmark suites [43,44,49,56]. Recent works [23,24,29,48] presented in-depth system performance characterization studies and shared the architectural implications for additional optimization opportunities. Our work here demonstrates a hardware-software co-design in action on deploying production-scale recommender systems to lowprecision hardware architectures.…”

Section: Related Workmentioning

confidence: 99%

Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale

et al. 2021

Self Cite

View full text Add to dashboard Cite

Tremendous success of machine learning (ML) and the unabated growth in ML model complexity motivated many ML-specific designs in both CPU and accelerator architectures to speed up the model inference. While these architectures are diverse, highly optimized low-precision arithmetic is a component shared by most. Impressive compute throughputs are indeed often exhibited by these architectures on benchmark ML models. Nevertheless, production models such as recommendation systems important to Facebook's personalization services are demanding and complex: These systems must serve billions of users per month responsively with low latency while maintaining high prediction accuracy, notwithstanding computations with many tens of billions parameters per inference. Do these low-precision architectures work well with our production recommendation systems? They do. But not without significant effort. We share in this paper our search strategies to adapt reference recommendation models to low-precision hardware, our optimization of low-precision compute kernels, and the design and development of tool chain so as to maintain our models' accuracy throughout their lifespan during which topic trends and users' interests inevitably evolve. Practicing these low-precision technologies helped us save datacenter capacities while deploying models with up to 5X complexity that would otherwise not be deployed on traditional general-purpose CPUs. We believe these lessons from the trenches promote better co-design between hardware architecture and software engineering and advance the state of the art of ML in industry.

show abstract

“…In particular, efficient training for the large embedding tables with varying memory access patterns imposes significant system design and optimization challenges. In addition, recent studies have started analyzing the system-and architecturelevel implications of neural recommendation inference [20], [21], [24]. Recent works also examine near memory processing architectures, such as RecNMP [29], TensorDIMM [30].…”

Section: Related Workmentioning

confidence: 99%

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

Acun

Murphy

Wang

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale datacenters, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components. GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.

show abstract

Cross-Stack Workload Characterization of Deep Recommendation Systems

Cited by 22 publications

References 38 publications

RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance

RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance

Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

Contact Info

Product

Resources

About