PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems

Abstract

The development of personalized recommendation has significantly improved the accuracy of information matching and the revenue of e-commerce platforms. Recently, it has two trends: 1) recommender systems must be trained timely to cope with ever-growing new products and ever-changing user interests from online marketing and social network; 2) state-of-the-art recommendation models introduce deep neural network (DNN) modules to improve prediction accuracy. Traditional CPU-based recommender systems cannot meet these two trends, and GPUcentric training has become a trending approach. However, we observe that GPU devices in training recommender systems are underutilized, and they cannot attain an expected throughput improvement as what it has achieved in Computer Vision (CV) and Neural Language Processing (NLP) areas. This issue can be explained by two characteristics of these recommendation models: First, they contain up to a thousand of input feature fields, introducing fragmentary and memory-intensive operations; Second, the multiple constituent feature interaction submodules introduce substantial small-sized compute kernels. To remove this roadblock to the development of recommender systems, we propose a novel framework named PICASSO to accelerate the training of recommendation models on commodity hardware. Specifically, we conduct a systematic analysis to reveal the bottlenecks encountered in training recommendation models. We leverage the model structure and data distribution to unleash the potential of hardware through our packing, interleaving, and caching optimization. Experiments show that PICASSO increases the hardware utilization by an order of magnitude on the basis of state-of-the-art baselines and brings up to 6× throughput improvement for a variety of industrial recommendation models. Using the same hardware budget in production, PICASSO on average shortens the walltime of daily training tasks by 7 hours, significantly reducing the delay of continuous delivery.

G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems

“…Parameter Server [9,20] is one of the most popular architectures for the distributed DLRM training and DMAML [5] customizes the Parameter Server architecture for MAML training in the CPU cluster. However, two update loops in meta learning double the computing and the computation-intensive dense layer becomes more complicated in DLRM [36,38], which makes the CPU time-consuming to compute and requires GPU for acceleration. Nevertheless, Parameter Server is mainly used in the CPU cluster and the design underutilizes the capability of GPU since the embedding layers held in servers are I/O and communicationintensive operators [30].…”

“…Secondly, meta learning requires different data management against the traditional deep learning training, and conventional I/O design bottlenecks the training speed [2,24]. Meta learning requires assembling the batch data according to both the task level and batch level when traditional deep learning only requires batch level in the training pipeline [1,36,38]. To be more specific, each worker may hold the batch data from different tasks, but the samples in a batch should belong to the identical task after shuffling for correctness.…”