Processing Big Data with Apache Hadoop in the Current Challenging Era of COVID-19

Abstract

Big data have become a global strategic issue, as increasingly large amounts of unstructured data challenge the IT infrastructure of global organizations and threaten their capacity for strategic forecasting. As experienced in former massive information issues, big data technologies, such as Hadoop, should efficiently tackle the incoming large amounts of data and provide organizations with relevant processed information that was formerly neither visible nor manageable. After having briefly recalled the strategic advantages of big data solutions in the introductory remarks, in the first part of this paper, we focus on the advantages of big data solutions in the currently difficult time of the COVID-19 pandemic. We characterize it as an endemic heterogeneous data context; we then outline the advantages of technologies such as Hadoop and its IT suitability in this context. In the second part, we identify two specific advantages of Hadoop solutions, globality combined with flexibility, and we notice that they are at work with a “Hadoop Fusion Approach” that we describe as an optimal response to the context. In the third part, we justify selected qualifications of globality and flexibility by the fact that Hadoop solutions enable comparable returns in opposite contexts of models of partial submodels and of models of final exact systems. In part four, we remark that in both these opposite contexts, Hadoop’s solutions allow a large range of needs to be fulfilled, which fits with requirements previously identified as the current heterogeneous data structure of COVID-19 information. In the final part, we propose a framework of strategic data processing conditions. To the best of our knowledge, they appear to be the most suitable to overcome COVID-19 massive information challenges.

Data-Driven Analytics Leveraging Artificial Intelligence in the Era of COVID-19: An Insightful Review of Recent Developments

“…For example, when a sporadic cluster emerged at a gay club, then CCTV data has played a vital role to identify the people who may have been come into contact with infected people leveraging multiple CCTVs footage. We invite interested readers to gain more insight about type and nature of data processed in EDLC in previous studies [59,60]. In [59], authors discussed the consequences of COVID-19 on different people based on their working environments.…”

“…In [59], authors discussed the consequences of COVID-19 on different people based on their working environments. In [60], authors discussed the importance of big data technologies in processing large scale data. Most of the AI models can handle the missing values present in a data.…”

Opportunities and Challenges in the Use of Big Data in Healthcare: A Literature Review

“…The use of BD is relevant, especially in unforeseen situations, such as the current health emergency of Covid-19 [1,30]. Following initial difficulty in containing the emergency, high-tech technologies, such as BDA and AI, have played important roles in the battle against Covid-19 [10,14,61]. For example, advanced analysis techniques, such as monitoring people's movements, understanding health trends and managing and administering drugs, have been developed to predict the spread of the virus and reduce the number of positive patients [6,14,62].…”

“…To support the growing volume of data available and overcome the inability to store and analyse it using traditional databases, new technologies have been developed that can handle its processing and provide insights for decision making [7][8][9][10][11]. The growing focus on BD has led several organisations to invest heavily in implementing the same [3,12].…”

Switching Towards a Proactive Grid Based Data Management Approach

“…Based on the MapReduce architecture, the data processing layer controls the parallelization of calculations throughout the cluster [8]. As a result, Hadoop has two types of Masters: the NameNode, which is in charge of the HDFS component, and the JobTracker, which is in charge of MapReduce implementation [10]. The NameNode is replicated for high availability purposes and the remainder of the cluster comprises nodes running the storage process DataNode (communicates with HDFS exclusively NameNode) and TaskTraker for processing of data (communicates exclusively with YARN JobTracker or resource manager) [11].…”