MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance

Mattson, Peter; Tang, Hanlin; Wei, Gu-Yeon; Wu, Carole-Jean; Reddi, Vijay Janapa; Cheng, Christine; Coleman, Cody; Diamos, Greg; Kanter, David; Micikevicius, Paulius; Patterson, David A.; Schmuelling, Guenther

doi:10.1109/mm.2020.2974843

Cited by 136 publications

(74 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The code is made available on GitHub. 2 The proposed settings match the globally optimal performance and outperform the suggested settings from Intel [Anju 2018] and TensorFlow [Google 2019] performance guides by 1.30× and 1.38×, respectively, across a set of real-world DL models, including several from the MLPerf suite [Mattson et al 2020].…”

Section: Introductionmentioning

confidence: 80%

“…Workloads. We use a set of production-size deep learning models, including three from MLPerf [Mattson et al 2020] (ResNet-50 [He et al 2016], Transformer [Vaswani et al 2017], neural collaborative filtering (NCF) ), as well as DenseNet [Huang et al 2017], SqueezeNet [Iandola et al 2016], Inception [Szegedy et al 2016], GoogLeNet [Szegedy et al 2015], CaffeNet [Jia et al 2014], ResNext [Xie et al 2017], and Google's Wide & Deep Learning model [Cheng n.d.]. To deeply understand the design features, we use micro-benchmarks such Fig.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Exploiting Parallelism Opportunities with Deep Learning Frameworks

Wang

et al. 2020

ACM Trans. Archit. Code Optim.

Self Cite

View full text Add to dashboard Cite

State-of-the-art machine learning frameworks support a wide variety of design features to enable a flexible machine learning programming interface and to ease the programmability burden on machine learning developers. Identifying and using a performance-optimal setting in feature-rich frameworks, however, involves a non-trivial amount of performance profiling efforts and often relies on domain-specific knowledge. This article takes a deep dive into analyzing the performance impact of key design features in a machine learning framework and quantifies the role of parallelism. The observations and insights distill into a simple set of guidelines that one can use to achieve much higher training and inference speedup. Across a diverse set of real-world deep learning models, the evaluation results show that the proposed performance tuning guidelines outperform the Intel and TensorFlow recommended settings by 1.30× and 1.38×, respectively.

show abstract

Section: Introductionmentioning

confidence: 80%

Section: Methodsmentioning

confidence: 99%

Exploiting Parallelism Opportunities with Deep Learning Frameworks

Wang

et al. 2020

ACM Trans. Archit. Code Optim.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In future work, we plan to assess the performance of the DONN on state-of-the-art DNN workloads, such as the models described in MLPerf 66 . Firstly, we will benchmark the DONN against all-electronic state-of-the-art accelerators by using Timeloop 67 .…”

Section: Discussionmentioning

confidence: 99%

Freely scalable and reconfigurable optical hardware for deep learning

Bernstein

Sludds

Hamerly

et al. 2021

Sci Rep

View full text Add to dashboard Cite

As deep neural network (DNN) models grow ever-larger, they can achieve higher accuracy and solve more complex problems. This trend has been enabled by an increase in available compute power; however, efforts to continue to scale electronic processors are impeded by the costs of communication, thermal management, power delivery and clocking. To improve scalability, we propose a digital optical neural network (DONN) with intralayer optical interconnects and reconfigurable input values. The path-length-independence of optical energy consumption enables information locality between a transmitter and a large number of arbitrarily arranged receivers, which allows greater flexibility in architecture design to circumvent scaling limitations. In a proof-of-concept experiment, we demonstrate optical multicast in the classification of 500 MNIST images with a 3-layer, fully-connected network. We also analyze the energy consumption of the DONN and find that digital optical data transfer is beneficial over electronics when the spacing of computational units is on the order of $$>10\,\upmu $$ > 10 μ m.

show abstract

“…When it comes to machine learning innovation in academia, one of the most significant assets can be large and open data sets that the community can use to prototype and quantitatively compare techniques in ways that show better value in terms of speed, accuracy, or implementation ease. This statement is supported by the significant efforts in time-series data classification 3 , image recognition 4 , and the larger machine learning community in general, both hardware and software 5 .…”

Section: Background and Summarymentioning

confidence: 94%

The Building Data Genome Project 2, energy meter data from the ASHRAE Great Energy Predictor III competition

et al. 2020

View full text Add to dashboard Cite

This paper describes an open data set of 3,053 energy meters from 1,636 non-residential buildings with a range of two full years (2016 and 2017) at an hourly frequency (17,544 measurements per meter resulting in approximately 53.6 million measurements). These meters were collected from 19 sites across North America and Europe, with one or more meters per building measuring whole building electrical, heating and cooling water, steam, and solar energy as well as water and irrigation meters. Part of these data was used in the Great Energy Predictor III (GEPIII) competition hosted by the American Society of Heating, Refrigeration, and Air-Conditioning Engineers (ASHRAE) in October-December 2019. GEPIII was a machine learning competition for long-term prediction with an application to measurement and verification. This paper describes the process of data collection, cleaning, and convergence of time-series meter data, the meta-data about the buildings, and complementary weather data. This data set can be used for further prediction benchmarking and prototyping as well as anomaly detection, energy analysis, and building type classification.

show abstract

MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance

Cited by 136 publications

References 2 publications

Exploiting Parallelism Opportunities with Deep Learning Frameworks

Exploiting Parallelism Opportunities with Deep Learning Frameworks

Freely scalable and reconfigurable optical hardware for deep learning

The Building Data Genome Project 2, energy meter data from the ASHRAE Great Energy Predictor III competition

Contact Info

Product

Resources

About