Real-time pneumonia prediction using pipelined spark and high-performance computing

Abstract

Background Pneumonia is a respiratory disease caused by bacteria; it affects many people, particularly in impoverished countries where pollution, unclean living standards, overpopulation, and insufficient medical infrastructures are prevalent. To guarantee curative therapy and boost survival chances, it is vital to detect pneumonia soon enough. Imaging using chest X-rays is the most common way of detecting pneumonia. However, analyzing chest X-rays is a complex process vulnerable to subjective variation. Moreover, the data available is growing exponentially, and it will take hours and days to train the model to predict pneumonia. Timely prediction is significant to guarantee a better cure and treatment. Existing work provided by different authors needs more precision, and the computation time for predicting pneumonia is also much longer. Therefore, there is a requirement for early forecasting. Using X-ray picture samples, the system must have a continuous and unsupervised learning system for early diagnosis. Methods In this article, the training time of the model is accelerated using the distributed data-parallel approach and the computational power of high-performance computing devices. This research aims to diagnose pneumonia using X-ray pictures with more precision, greater speed, and fewer processing resources. Distributed deep learning techniques are gaining popularity owing to the rising need for computational resources for deep learning models with several parameters. In contrast to conventional training methods, data-parallel training enables several compute nodes to train massive deep-learning models to improve training efficiency concurrently. Deploying the model in Spark solves the scalability and acceleration. Spark’s distributed processing capability reads data from multiple nodes, and the results demonstrate that training time can be drastically reduced by utilizing these techniques, which is a significant necessity when dealing with large datasets. Results The proposed model makes the prediction 1.5 times faster than the traditional CNN model used for pneumonia prediction. The model also achieved an accuracy of 98.72%. The speed-up varying from 1.2 to 1.5 was obtained in the synchronous and asynchronous parallel model. The speed-up is reduced in the parallel asynchronous model due to the presence of straggler nodes.

Health Care Equity Through Intelligent Edge Computing and Augmented Reality/Virtual Reality: A Systematic Review

“… Edge computing health model using P2P-based deep neural networks. 19 Deep learning algorithms are hampered by the over-fitting issue in a neural network and increased computing costs related to elevated levels of time complexity. Response delays are common in large data learning processes and deep neural network-based data extraction procedures because of these issues, which exponentially raise the cost of data extraction.…”

Computationally Efficient Neural Rendering for Generator Adversarial Networks Using a Multi-GPU Cluster in a Cloud Environment

“…The single-node and multi-node acceleration on CNN and LSTM was deeply analyzed for different use cases but was not expanded to GAN [36], [37]. The impact of the multinode on spark and GPU was examined for medical use cases [35], [38]. The impact of multi-node TPU on GAN for double precision was developed but lacked deployment and did not address the current bottleneck issues in TPU [34].…”

Dynamic Clustering Strategies Boosting Deep Learning in Olive Leaf Disease Diagnosis

“…This highlights the computational challenges of train-ing CNNs. In addition, deploying and training CNNs requires significant computational resources, such as powerful GPUs or specialized equipment [28,29].…”

“…Furthermore, the combination of dynamic clustering and deep learning solves the challenges of large datasets. By partitioning the data into evolving clusters, computational efficiency is increased, convergence is accelerated, and memory requirements during training are reduced [28]. The synergy between dynamic clustering and deep learning enables models to exploit data variability effectively, adapt to changing patterns, and achieve high-performance benchmarks in various applications, including image recognition, natural language processing, and recommender systems.…”