M3r

Shinnar, Avraham; Cunningham, David; Saraswat, Vijay; Herta, Benjamin

doi:10.14778/2367502.2367513

Cited by 104 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These algorithms are typically iterative; thus, in-memory computing suits well the need for cached data to be reused. Similarly, pure Map-Reduce paradigms have benefited from in-memory trends, resulting in platforms for memory-intensive workloads such as Mammoth [36], Piccolo [54], Main-Memory Map-Reduce (M3R) [55], and Hyracks [56].…”

Section: B Data-centric Batch and Stream Processingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

Section: B Data-centric Batch and Stream Processingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Some methods use In-Memory techniques to decrease the Map-Reduce job execution time. PowerDrill [43], Shark [44], Spark and M3R [45] are prominent methods. Pow-erDrill is a column-based method.…”

Section: Related Workmentioning

confidence: 99%

Chabok: a Map-Reduce based method to solve data warehouse problems

Barkhordari

Niamanesh

2018

J Big Data

View full text Add to dashboard Cite

Existing information is a valuable asset for many different types of organizations. Storing and analysing information can solve many problems within an organization [1]. The results from data analyses help organizations make correct decisions and provide better services for customers. Thus, high speed storage and retrieval of large volumes of data generated by electronic devices and software systems are critical issues [2-4]. Many organizations consider big data solutions because they cannot manage their data with traditional database management systems [5]; therefore, they must seek drastic measures for the design and implementation of new systems according to big data architectures. These organizations must change their architectures from Abstract Currently, immense quantities of data cannot be managed by traditional database management systems. Instead, they must be managed by big data solutions using shared nothing architectures. Data warehouse systems are systems that address very large amounts of information. The most prominent data warehouse model is star schema, which consists of a fact table and some number of dimension tables. It is necessary to join the facts and dimensions for query executions on the data warehouse. In shared nothing architecture, all of the required information is not placed on a single node so it is necessary to retrieve information from other nodes, which causes network congestion and low speeds of query execution. To avoid this problem and achieve maximum parallelism, dimensions can be replicated over nodes if they are not too large. However, if there are dimensions with data volumes greater than the capacity of a node or dimensions where the data volume summation exceeds node capacity, the query execution is confronted with serious problems. In big data problems, the amount of data is immense, and thus replicating immense data cannot be considered an appropriate method. In this paper, we propose a method called Chabok, which uses two-phased Map-Reduce to solve the data warehouse problem. In this method, aggregation is performed completely on Mappers, and intermediate results are sent to the Reducer. Chabok does not need data replication for join omission. The proposed method was implemented on Hadoop, and TPC-DS queries were executed for benchmarking. The query execution time on Chabok surpassed prominent big data products for data warehousing.

show abstract

“…However, these studies are still based on the batch processing for static data increment, and their performance improvements are limited when processing continuous high-speed data streams. S4 [10], M3R [16], Storm [17] and HOP [2] can achieve real-time data stream processing using MapReduce-like programming model by pipelining the application process and distributing data and computing logic to distributed nodes. But these studies only focus on streaming data processing, and lack direct supports, e.g., historical data pre-processing and caching, of historical data processing.…”

Section: B Related Workmentioning

confidence: 99%

A Hybrid Processing System for Large-Scale Traffic Sensor Data

et al. 2015

View full text Add to dashboard Cite

In recent years, with the further adoption of the Internet of Things and sensor technology, all kinds of intelligent transportation system (ITS) applications based on a wide range of traffic sensor data have had rapid development. Traffic sensor data gathered by large amounts of sensors show some new features, such as massiveness, continuity, streaming, and spatio-temporality. ITS applications utilizing traffic sensor data can be divided into three main types: 1) offline processing of historical data; 2) online processing of streaming data; and 3) hybrid processing of both. Current research tends to solve these problems in separate solutions, such as stream computing and batch processing. In this paper, we propose a hybrid processing approach and present corresponding system implementation for both streaming and historical traffic sensor data, which combines spatio-temporal data partitioning, pipelined parallel processing, and stream computing techniques to support hybrid processing of traffic sensor data in real-time. Three types of real-world applications are explained in detail to show the usability and generality of our approach and system. Our experiments show that the system can achieve better performance than a popular open-source streaming system called Storm.INDEX TERMS Traffic sensor data, spatio-temporal data object, real-time processing, stream computing.

show abstract

M3r

Cited by 104 publications

References 3 publications

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Chabok: a Map-Reduce based method to solve data warehouse problems

A Hybrid Processing System for Large-Scale Traffic Sensor Data

Contact Info

Product

Resources

About