Building High-throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine

Abstract

Neural networks (NN) are used in high-performance computing and high-throughput analysis to extract knowledge from datasets. Neural architecture search (NAS) automates NN design by generating, training, and analyzing thousands of NNs. However, NAS requires massive computational power for NN training. To address challenges of efficiency and scalability, we propose PENGUIN, a decoupled fitness prediction engine that informs the search without interfering in it. PENGUIN uses parametric modeling to predict fitness of NNs. Existing NAS methods and parametric modeling functions can be plugged into PENGUIN to build flexible NAS workflows. Through this decoupling and flexible parametric modeling, PENGUIN reduces training costs: it predicts the fitness of NNs, enabling NAS to terminate training NNs early. Early termination increases the number of NNs that fixed compute resources can evaluate, thus giving NAS additional opportunity to find better NNs. We assess the effectiveness of our engine on 6,000 NNs across three diverse benchmark datasets and three state of the art NAS implementations using the Summit supercomputer. Augmenting these NAS implementations with PENGUIN can increase throughput by a factor of 1.6 to 7.1. Furthermore, walltime tests indicate that PENGUIN can reduce training time by a factor of 2.5 to 5.3.

High frequency accuracy and loss data of random neural networks trained on image datasets

“…IEEE Transactions on Parallel and Distributed Systems, 2022, In Press. DOI 10.1109/TPDS.2022.3140681 [12] …”

“…Such tools include neural architecture search [6] , [7] , [8] and methods for NN fitness prediction and training termination [9] , [10] , [11] . Learning curve data is essential to the development of methods for NN fitness modeling and prediction [3] , [12] . Researchers can use our dataset to study evolution of NN fitness during training and identify relationships between an NN’s structure and its fitness on a given image dataset.…”

“…Statistical methods can be applied to our dataset to analyze the shape of the learning curves. This enables researchers to identify families of functions that well model such curves and make informed choices about which modeling functions to employ in parametric modeling methods [12] . Our data can advance effective searches for accurate NNs, which have a far-reaching impact on many fields.…”

“…Such tools include neural architecture search [6] , [7] , [8] and methods for NN fitness prediction and training termination [9] , [10] , [11] . Learning curve data is essential to the development of methods for NN fitness modeling and prediction [3] , [12] .…”