A Survey on Auto-Parallelism of Large-Scale Deep Learning Training

Liang, Peng; Tang, Yan; Zhang, Xiaoda; Bai, Youhui; Su, Teng; Lai, Zhiquan; Qiao, Linbo; Li, Dongsheng

doi:10.1109/tpds.2023.3281931

Cited by 9 publications

(2 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The open-source movement has democratized access to cutting-edge tools, with platforms such as TensorFlow [38] and PyTorch [39] leading the way. This democratization has led to the emergence of MLOps, a practice that seeks to standardize and automate the AI model lifecycle [40,41].…”

Section: Open-source Architectures and Mlopsmentioning

confidence: 99%

Fostering Agricultural Transformation through AI: An Open-Source AI Architecture Exploiting the MLOps Paradigm

Cob-Parro,

Lalangui,

Lazcano

2024

Agronomy

View full text Add to dashboard Cite

As the global population is expected to reach 10 billion by 2050, the agricultural sector faces the challenge of achieving an increase of 60% in food production without using much more land. This paper explores the potential of Artificial Intelligence (AI) to bridge this “land gap” and mitigate the environmental implications of agricultural land use. Typically, the problem with using AI in such agricultural sectors is the need for more specific infrastructure to enable developers to design AI and ML engineers to deploy these AIs. It is, therefore, essential to develop dedicated infrastructures to apply AI models that optimize resource extraction in the agricultural sector. This article presents an infrastructure for the execution and development of AI-based models using open-source technology, and this infrastructure has been optimized and tuned for agricultural environments. By embracing the MLOps culture, the automation of AI model development processes is promoted, ensuring efficient workflows, fostering collaboration among multidisciplinary teams, and promoting the rapid deployment of AI-driven solutions adaptable to changing field conditions. The proposed architecture integrates state-of-the-art tools to cover the entire AI model lifecycle, enabling efficient workflows for data scientists and ML engineers. Considering the nature of the agricultural field, it also supports diverse IoT protocols, ensuring communication between sensors and AI models and running multiple AI models simultaneously, optimizing hardware resource utilization. Surveys specifically designed and conducted for this paper with professionals related to AI show promising results. These findings demonstrate that the proposed architecture helps close the gap between data scientists and ML engineers, easing the collaboration between them and simplifying their work through the whole AI model lifecycle.

show abstract

Section: Open-source Architectures and Mlopsmentioning

confidence: 99%

Fostering Agricultural Transformation through AI: An Open-Source AI Architecture Exploiting the MLOps Paradigm

Cob-Parro,

Lalangui,

Lazcano

2024

Agronomy

View full text Add to dashboard Cite

show abstract

“…Recently, in order to eliminate the complexity of model parallelism, Auto Parallelism is being studied. Auto Parallelism automatically implements model parallelism by considering the hardware and model structure at the system level [22]. This shows novel parallelism performance in all general-purpose deep-learning models, but it is necessary to design model parallelism directly to secure resource use efficiency in large-scale artificial intelligence models.…”

Section: Related Workmentioning

confidence: 99%

Optimizing Single DGX-A100 System: Overcoming GPU Limitations via Efficient Parallelism and Scheduling for Large Language Models

Kim

Jeong

2023

Applied Sciences

View full text Add to dashboard Cite

In this study, we introduce a novel training algorithm specifically designed to overcome the limitations of GPU memory on a single DGX-A100 system. By utilizing the CPU and main memory in the training process and applying a strategy of division and parallelization, our algorithm enhances the size of the trainable language model and the batch size. In addition, we developed a comprehensive management system to effectively manage the execution of the algorithm. This system systematically controls the training process and resource usage, while also enabling the asynchronous deployment of tasks. Finally, we proposed a scheduling technique integrated into the management system, promoting efficient task scheduling in a complex, heterogeneous training environment. These advancements equip researchers with the ability to work with larger models and batch sizes, even when faced with limited GPU memory.

show abstract