Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Abstract

Distributed computing frameworks are the fundamental component of distributed computing systems.They provide an essential way to support the efficient processing of big data on clusters or cloud. The size of big data increases at a pace that is faster than the increase in the big data processing capacity of clusters. Thus, distributed computing frameworks based on the MapReduce computing model are not adequate to support big data analysis tasks which often require running complex analytical algorithms on extremely big data sets in terabytes.In performing such tasks, these frameworks face three challenges: computational inefficiency due to high I/O and communication costs, non-scalability to big data due to memory limit, and limited analytical algorithms because many serial algorithms cannot be implemented in the MapReduce programming model. New distributed computing frameworks need to be developed to conquer these challenges. In this paper, we review MapReduce-type distributed computing frameworks that are currently used in handling big data and discuss their problems when conducting big data analysis. In addition, we present a non-MapReduce distributed computing framework that has the potential to overcome big data analysis challenges.

Research on Student Management Platform Based on Big Data Under Low-Carbon Environment

“…Its underlying core is Hadoop Distributed File System (HDFS) and the MapReduce framework, each of which has different functions. Sun et al (2023) analyzed three challenges in a distributed computing framework based on the MapReduce computing model: 1) computational inefficiency due to high I/O and communication costs, 2) inability to scale to big data due to memory constraints, and 3) inability to scale to big data due to the fact that many serial algorithms cannot be implemented in the MapReduce programming model. They finally proposed a non-MapReduce distributed computing framework, which has the potential to overcome the challenges of big data analytics.…”

A Novel Memory Concurrent Editing Model for Large-Scale Video Streams in Edge Computing

“…That is, some are busy, and some are idle, but the distribution and merging of data are imperceptible to other editors. Borrowing the thinking of MapReduce distributed/parallel computing [6], there are other editors who can queue up to enter the editing area [7]. The concurrent editing framework for video streaming is shown in Figure 1.…”