Map-Reduce Implementations: Survey and Performance Comparison

Khanam, Zeba; Agarwal, Shafali

doi:10.5121/ijcsit.2015.7410

Cited by 10 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data is nowadays stored and processed in distributed systems, often in a geographically dispersed manner. This introduces a layer of complexity that MapReduce frameworks, such as Apache Spark, excel at handling [61]. Container engines, and in particular Docker, are also becoming an essential part of bioinformatics pipelines as they improve delivery, interoperability and reproducibility of scientific analyses.…”

Section: Discussionmentioning

confidence: 99%

MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

Capuccini¹,

Dahlö²,

Toor³

et al. 2018

Preprint

View full text Add to dashboard Cite

Background. Life science is increasingly driven by Big Data analytics, and the MapReduce programming model has been proven successful for data-intensive analyses. However, current MapReduce frameworks offer poor support for reusing existing processing tools in bioinformatics pipelines. Further, these frameworks do not have native support for application containers, which are becoming popular in scientific data processing.Results. Here we present MaRe, a programming model with an associated open-source implementation, which introduces support for application containers in MapReduce. MaRe is based on Apache Spark and Docker, the MapReduce framework and container engine that have collected the largest open source community, thus providing interoperability with the cutting-edge software ecosystem. We demonstrate MaRe on two data-intensive applications in life science, showing ease of use and scalability.Conclusions. MaRe enables scalable data-intensive processing in life science with MapReduce and application containers. When compared with current best practices, that involve the use of workflow systems, MaRe has the advantage of providing data locality, ingestion from heterogeneous storage systems and interactive processing. MaRe is generally-applicable and available as open source software.1 Findings Background and purposeLife science is increasingly driven by Big Data analytics. From genomics, proteomics and metabolomics to bioimaging and drug discovery, scientists need to analyze larger and larger amounts of data [1,2,3,4,5]. This means that datasets can no longer be stored and processed in a researcher's workstation, but they instead need to be handled on distributed systems, at organization level. For instance, the European Bioinformatics Institute, in Hinxton (United Kingdom), offers a total storage capacity of over 160 petabytes for biologically-significant data [6]. Such amounts of data poses major challenges for scientific analyses. First, there is a need to efficiently scale existing processing tools over massive datasets. In fact, bioinformatics software that was originally developed with the simplistic view of small-scale data, will not scale on distributed computing platforms out of the box. The process of adapting such tools may introduce disruptive changes to the existing codebase, and it is generally unsustainable for most organizations. Secondly, the

show abstract

Section: Discussionmentioning

confidence: 99%

MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

Capuccini¹,

Dahlö²,

Toor³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The first one is a data storage called Hadoop DFS (HDFS), whereas the other is an information process called (Hadoop MapReduce Framework). Hadoop DFS is a block-structured filing system controlled by only one master node as Google's GFS [18].…”

Section: A Volume: This Criterion Represents the Most Immedi-mentioning

confidence: 99%

Predicting Heart Diseases from Large Scale IoT Data Using a Map-Reduce Paradigm

Mohammad

Manaa

2020

Open Computer Science

View full text Add to dashboard Cite

Over the last few years, the huge amount of data represented a major obstacle to data analysis. Big data implies that the volume of data undergoes a faster progress than computational speeds, thereby demanding a larger data storage capacity. The Internet of Things (IoT) is a main source of data that is closely related to big data, as the former extends to a variety of fields such as healthcare, entertainment, and disaster control. Despite the different advantages associated with the composition of Big Data analytics and IoT, there are a number of complex difficulties and issues involved that need to be resolved and managed to ensure an accurate data analysis. Some of these solutions include the utilization of map-reduce techniques, processing, and large data scale, particularly for the relatively less time that this method requires to process large data from the Internet of Things. Machine learning algorithms of this kind are often implemented in the healthcare sector. Medical facilities need to be advanced so that more appropriate decisions can be made in terms of patient diagnosis and treatment options. In this work, two datasets have been used: the first set, used in the prediction of heart diseases, obtained an accuracy rate of 84.5 for RF and 83 for J48, whereas the second dataset is related to weather stations (automated sensors) and obtained accuracy rates of 88.5 and 86.5 for RF and J48, respectively.

show abstract

“…The authors Zeba Khanam and Shafali Agarwal, in their paper [3], explore large scale data processing using MapReduce and its various implementations to facilitate the database, researchers and other communities in developing the technical understanding of the MapReduce framework. They continue with exploring different MapReduce implementations; most popular Hadoop implementations and other similar implementations using other platforms and compare those based on different parameters.…”

Section: Related Workmentioning

confidence: 99%

MapReduce Performance in MongoDB Sharded Collections

Ajdari¹,

Kasami²

2018

ijacsa

View full text Add to dashboard Cite

In the modern era of computing and countless of online services that gather and serve huge data around the world, processing and analyzing Big Data has rapidly developed into an area of its own. In this paper, we focus on the MapReduce programming model and associated implementation for processing and analyzing large datasets in a NoSQL database such as MongoDB. Furthermore, we analyze the performance of MapReduce in sharded collections with huge dataset and we measure how the execution time scales when the number of shards increases. As a result, we try to explain when MapReduce is an appropriate processing technique in MongoDB and also to give some measures and alternatives to take when MapReduce is used.

show abstract

Map-Reduce Implementations: Survey and Performance Comparison

Cited by 10 publications

References 10 publications

MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

Predicting Heart Diseases from Large Scale IoT Data Using a Map-Reduce Paradigm

MapReduce Performance in MongoDB Sharded Collections

Contact Info

Product

Resources

About