Fathom: reference workloads for modern deep learning methods

Abstract

Deep learning has been popularized by its recent successes on challenging artificial intelligence problems. One of the reasons for its dominance is also an ongoing challenge: the need for immense amounts of computational power. Hardware architects have responded by proposing a wide array of promising ideas, but to date, the majority of the work has focused on specific algorithms in somewhat narrow application domains. While their specificity does not diminish these approaches, there is a clear need for more flexible solutions. We believe the first step is to examine the characteristics of cutting edge models from across the deep learning community.Consequently, we have assembled Fathom: a collection of eight archetypal deep learning workloads for study. Each of these models comes from a seminal work in the deep learning community, ranging from the familiar deep convolutional neural network of Krizhevsky et al., to the more exotic memory networks from Facebook's AI research group. Fathom has been released online, and this paper focuses on understanding the fundamental performance characteristics of each model. We use a set of application-level modeling tools built around the TensorFlow deep learning framework in order to analyze the behavior of the Fathom workloads. We present a breakdown of where time is spent, the similarities between the performance profiles of our models, an analysis of behavior in inference and training, and the effects of parallelism on scaling.

Characterizing Deep Learning Training Workloads on Alibaba-PAI

“…In our cluster, commodity embedding, search and recommendation workloads have large training datasets and may exploit hundreds to thousands of workers to achieve high throughput on the huge training dataset. Notably, such extra large-scale workloads always have significant commercial impact on the company's business; however, they are often not included in DL workload benchmarks [16], [22]. We find that they are non-negligible: only 0.7% of all workloads have more than 128 cNodes; however, they consume more than 16% computation resource on our cluster.…”

“…Some other work aim to establish the performance benchmark [16], [18], [22], [50]. Fathom [16] establishes a set of reference implementation for eight archetypal DL jobs.…”

“…First, we present a lightweight framework to characterize the production workloads at the cluster level. We comprehensively include not only the basic aspects of computation and weight communication in training jobs, as considered in previous studies [16], [19], but also the input data I/O aspects. Our analysis shows that the data I/O time is nonnegligible, especially for single-node training workloads; for distributed workloads, input data I/O can potentially become the performance bottleneck after gradient communication has been optimized.…”

The Architectural Implications of Facebook's DNN-Based Personalized Recommendation

“…DNN performance analysis and optimization Current publicly available benchmarks [3,8,19,55] for DNNs fo-cus on neural networks with FC, CNN, and RNN layers only.…”

HPC AI500: A Benchmark Suite for HPC AI Systems

“…Most of the existing AI benchmarks [8,27,9,7,26,10] are based on commercial scenarios. Deep500 [28] is a benchmarking framework aiming to evaluate high-performance deep learning.…”