Efficient and Systematic Partitioning of Large and Deep Neural Networks for Parallelization

Wang, Haoran; Li, Chong; Tachon, Thibaut; Wang, Hongxing; Yang, Shi Chun; Limet, Sébastien; Robert, Sophie

doi:10.1007/978-3-030-85665-6_13

Cited by 8 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(iii) The operators are not executed separately. They are all connected in the computational graph via the edges [19]. The output tensor of an operator is also the input tensor of its successive operator.…”

Section: Distributed Strategies Searchingmentioning

confidence: 99%

“…5 A zipper structure representing the path highlighted of the binary tree methods are consequently suitable for coarser grained graph representation, whereas a finer-grained representation provides more possibilities. This work does not provide implementation details because a more practical point of view was already given in previous work [19].…”

Section: From Tree To Graphmentioning

confidence: 99%

“…Searching Algorithm: SMSG offers the ability to model the cost and compute it with homomorphism. As for the searching algorithm, we implemented a linear-complexity searching algorithm D-Rec [19] for the experiments in this section.…”

Section: Dnn Modelsmentioning

confidence: 99%

See 2 more Smart Citations

SMSG: Profiling-Free Parallelism Modeling for Distributed Training of DNN

Wang

Tachon

et al. 2022

Int J Parallel Prog

Self Cite

View full text Add to dashboard Cite

The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for DNN designers. Some automatic searching approaches have recently been studied to free the experts from the heavy parallel strategy conception. However, these approaches all rely on a numerical cost model, which requires heavy profiling results that lack portability. These profiling-based approaches cannot lighten the strategy generation work due to the non-reusable profiling value. Our intuition is that there is no need to estimate the actual execution time of the distributed training but to compare the relative cost of different strategies. We propose SMSG (Symbolic Modeling for Strategy Generation), which analyses the cost based on the communication and computation semantics. With SMSG, the parallel cost analyses are decoupled from hardware characteristics. SMSG defines cost functions for each kind of operator to quantitatively evaluate the amount of data for computation and communication, which eliminates the heavy profiling tasks. Besides, SMSG introduces how to apply functional transformation by using the Third Homomorphism theorem to control the high searching complexity. Our experiments show that SMSG can find good hybrid parallelism strategies to generate an efficient training performance similar to the state of the art. Moreover, SMSG covers a wide variety of DNN models with good scalability. SMSG provides good portability when changing training configurations that a profiling-based approach cannot.

show abstract

Section: Distributed Strategies Searchingmentioning

confidence: 99%

Section: From Tree To Graphmentioning

confidence: 99%