Communication-Efficient Distributed Deep Learning with GPU-FPGA Heterogeneous Computing

Tanaka, Kenji; Arikawa, Yuki; Ito, Tsuyoshi; Morita, Kazutaka; Nemoto, Naru; Miura, Fumiaki; Terada, Kazuhiko; Teramoto, Junji; Sakamoto, Takeshi

doi:10.1109/hoti51249.2020.00021

Cited by 7 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning has been proved to be incredibly effective in a number of practical applications of artificial intelligence [24]. This is despite the fact that it demands a significant amount of computer resources in order to train a model such that it is of a high quality.…”

Section: Gpu Clustermentioning

confidence: 99%

“…The researchers that carried out the study [24]came to the conclusion that coupled communication calculations, such as All reduction, which are used to split processing outputs among graphics processing units (GPUs), create an unavoidable bottleneck. This is in conformity with the conclusions of the researchers.…”

Section: Literature Reviewmentioning

confidence: 99%

“…In [24], when deep learning is distributed over a large number of GPUs, there is always a communication barrier that has to be overcome. The significant difference in the amount of time it takes for data to be sent between the inter-router connection and the presentation router communication is the root cause of this bottleneck.…”

Section: Literature Reviewmentioning

confidence: 99%

See 2 more Smart Citations

Modular Platforms based on Clouded Web Technology and Distributed Deep Learning Systems

Abdullah,

Abdulrahman,

Abdulkareem

et al. 2023

Journal of Smart Internet of Things

View full text Add to dashboard Cite

Utilising the dispersed resources that are accessible inside a cluster, the dispersed Deep Learning System (DDLS) is able to successfully complete the process of training complex neural network models. This is accomplished by utilising the resources to their full potential. As a consequence of this, the system is able to get insights about neural network models that are becoming more complex. Individuals who work as engineers for distributed deep learning systems are needed to make a variety of decisions in order to properly manage their specialised job within the environment of their choice. This is necessary in order to ensure that the job is efficiently managed. Achieving this is very necessary in order to guarantee that they will be able to carry out their obligations. Throughout the course of the last several years, deep learning programmes have uncovered significant applications in a broad range of different industries. A few examples of these domains include image recognition, natural language processing, semantic understanding, financial analysis, and aided healthcare. These are only few of the topics that fall under this category. All of these factors have led to a significant growth in the amount of information that is being used in each and every application. The continued penetration of information into a number of different sectors, as well as the increasing complexity of computations and the restrictions of models, have all contributed to this. Because of this, there has been a significant increase in the quantity of information that is being employed on a worldwide scale for various purposes.

show abstract

Section: Gpu Clustermentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Modular Platforms based on Clouded Web Technology and Distributed Deep Learning Systems

Abdullah,

Abdulrahman,

Abdulkareem

et al. 2023

Journal of Smart Internet of Things

View full text Add to dashboard Cite

show abstract

“…Latency results from the time required to fetch model parameters from off-chip DRAM or external SDCARDs before appropriate computation can be performed on these parameters [ 150 ]. Thus, storing the parameters as close as possible to the computation unit using Tiling and data reuse, hardware-oriented direct memory access (DMA) optimization techniques would reduce the latency and thus, inform high computation speed [ 152 ]. In addition, because ML models require a high level of parallelism for efficient performance, throughput is a major issue.…”

Section: Challenges and Optimization Opportunities In Embedded Machine Learningmentioning

confidence: 99%

“…This architectural organization if not properly utilized, can result in latency concerns. DMAs are units that transfer data between the external memory and the on-chip buffers in the processing logic of the FPGA [ 152 ]. Thus, optimizing this process would lead to an efficient performance in execution speed.…”

Section: Challenges and Optimization Opportunities In Embedded Machine Learningmentioning

confidence: 99%

An Overview of Machine Learning within Embedded and Mobile Devices–Optimizations and Applications

Ajani

Imoize

Atayero

2021

Sensors

105

View full text Add to dashboard Cite

Embedded systems technology is undergoing a phase of transformation owing to the novel advancements in computer architecture and the breakthroughs in machine learning applications. The areas of applications of embedded machine learning (EML) include accurate computer vision schemes, reliable speech recognition, innovative healthcare, robotics, and more. However, there exists a critical drawback in the efficient implementation of ML algorithms targeting embedded applications. Machine learning algorithms are generally computationally and memory intensive, making them unsuitable for resource-constrained environments such as embedded and mobile devices. In order to efficiently implement these compute and memory-intensive algorithms within the embedded and mobile computing space, innovative optimization techniques are required at the algorithm and hardware levels. To this end, this survey aims at exploring current research trends within this circumference. First, we present a brief overview of compute intensive machine learning algorithms such as hidden Markov models (HMM), k-nearest neighbors (k-NNs), support vector machines (SVMs), Gaussian mixture models (GMMs), and deep neural networks (DNNs). Furthermore, we consider different optimization techniques currently adopted to squeeze these computational and memory-intensive algorithms within resource-limited embedded and mobile environments. Additionally, we discuss the implementation of these algorithms in microcontroller units, mobile devices, and hardware accelerators. Conclusively, we give a comprehensive overview of key application areas of EML technology, point out key research directions and highlight key take-away lessons for future research exploration in the embedded machine learning domain.

show abstract

A Comprehensive Survey on SmartNICs: Architectures, Development Models, Applications, and Research Directions

Kfoury,

Choueiri,

Mazloum

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Communication-Efficient Distributed Deep Learning with GPU-FPGA Heterogeneous Computing

Cited by 7 publications

References 7 publications

Modular Platforms based on Clouded Web Technology and Distributed Deep Learning Systems

Modular Platforms based on Clouded Web Technology and Distributed Deep Learning Systems

An Overview of Machine Learning within Embedded and Mobile Devices–Optimizations and Applications

A Comprehensive Survey on SmartNICs: Architectures, Development Models, Applications, and Research Directions

Contact Info

Product

Resources

About