A study on using uncertain time series matching algorithms for MapReduce applications

Rizvandi, Nikzad Babaii; Taheri, Javid; Moraveji, Reza; Zomaya, Albert Y.

doi:10.1002/cpe.2895

Cited by 19 publications

(15 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MapReduce is a programming model for processing large datasets with a parallel, distributed algorithm on a computing cluster of low cost commodity computers. A MapReduce application typically consists of two phases (or operations); "map" and "reduce" with many tasks in each phase [83]. A map/reduce task deals with a chunk of data independently and thus, tasks in a given phase can be easily parallelized and effectively processed in a large-scale computing environment (i.e., a cloud platform).…”

Section: Mapreduce Parallel Processingmentioning

confidence: 99%

Big Data Analytics for Dynamic Energy Management in Smart Grids

2015

View full text Add to dashboard Cite

The smart electricity grid enables a two-way flow of power and data between suppliers and consumers in order to facilitate the power flow optimization in terms of economic efficiency, reliability and sustainability. This infrastructure permits the consumers and the micro-energy producers to take a more active role in the electricity market and the dynamic energy management (DEM). The most important challenge in a smart grid (SG) is how to take advantage of the users' participation in order to reduce the cost of power. However, effective DEM depends critically on load and renewable production forecasting. This calls for intelligent methods and solutions for the real-time exploitation of the large volumes of data generated by a vast amount of smart meters. Hence, robust data analytics, high performance computing, efficient data network management, and cloud computing techniques are critical towards the optimized operation of SGs. This research aims to highlight the big data issues and challenges faced by the DEM employed in SG networks. It also provides a brief description of the most commonly used data processing methods in the literature, and proposes a promising direction for future research in the field.Comment: Published in ELSEVIER Big Data Researc

show abstract

Section: Mapreduce Parallel Processingmentioning

confidence: 99%

Big Data Analytics for Dynamic Energy Management in Smart Grids

2015

View full text Add to dashboard Cite

show abstract

“…Similarly, (Palanisamy et al, 2015) deals with optimizing the allocation of VMs executing MapReduce jobs in order to minimize the infrastructure cost in a cloud datacenter. In the same single-cloud scenario, the work (Rizvandi et al, 2013) focuses on the automatic estimation of MapReduce configuration parameters, while (Verma et al, 2011) proposes a resource allocation algorithm able to estimate the amount of resources required to meet MapReduce-specific performance goals. However, these models were not intended to address the challenges of the hybrid cloud scenario, which is a possible target environment for the provisioning of additional VMs in our system thanks to the underlying HyIaaS layer.…”

Section: Related Workmentioning

confidence: 99%

Process Mining Monitoring for Map Reduce Applications in the Cloud

Chesani

Ciampolini

Loreti

et al. 2016

Proceedings of the 6th International Conference on Cloud Computing and Services Science

View full text Add to dashboard Cite

The adoption of mobile devices and sensors, and the Internet of Things trend, are making available a huge quantity of information that needs to be analyzed. Distributed architectures, such as Map Reduce, are indeed providing technical answers to the challenge of processing these big data. Due to the distributed nature of these solutions, it can be difficult to guarantee the Quality of Service: e.g., it might be not possible to ensure that processing tasks are performed within a temporal deadline, due to specificities of the infrastructure or processed data itself. However, relaying on cloud infrastructures, distributed applications for data processing can easily be provided with additional resources, such as the dynamic provisioning of computational nodes. In this paper, we focus on the step of monitoring Map Reduce applications, to detect situations where resources are needed to meet the deadlines. To this end, we exploit some techniques and tools developed in the research field of Business Process Management: in particular, we focus on declarative languages and tools for monitoring the execution of business process. We introduce a distributed architecture where a logic-based monitor is able to detect possible delays, and trigger recovery actions such as the dynamic provisioning of further resources.

show abstract

“…Then a fingerprint based method is utilized to predict the performance of a new MapReduce application based on the studied applications. The idea of pattern matching was used in [12] to find the similarity between CPU time patters of a new application and applications in database. Then it was concluded that if two applications show high similarity for several setting of configuration parameters it is very likely their optimal values of configuration parameters also be the same.…”

Section: Related Workmentioning

confidence: 99%

“…Our benchmark applications are WordCount (used by leading researchers in Intel [21], IBM [6], MIT [22], and UC-Berkeley [7]), TeraSort (as a standard benchmark in the international TeraByte sort competition [23,24] as well as many researchers in IBM [25,26], Intel [21], INRIA [27] and UC-Berkeley [28]), and Exim Mainlog parsing [12,29]. These benchmarks are used due to their striking differences as well as their popularity among MapReduce applications.…”

Section: Experimental Settingmentioning

confidence: 99%

On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments

Rizvandi

Taheri

Moraveji

et al. 2012

Algorithms and Architectures for Parallel Processing

Self Cite

View full text Add to dashboard Cite

recently, businesses have started using MapReduce as a popular computation framework for processing large amount of data, such as spam detection, and different data mining tasks, in both public and private clouds. Two of the challenging questions in such environments are (1) choosing suitable values for MapReduce configuration parameters -e.g., number of mappers, number of reducers, and DFS block size-, and (2) predicting the amount of resources that a user should lease from the service provider. Currently, the tasks of both choosing configuration parameters and estimating required resources are solely the users' responsibilities. In this paper, we present an approach to provision the total CPU usage in clock cycles of jobs in MapReduce environment. For a MapReduce job, a profile of total CPU usage in clock cycles is built from the job past executions with different values of two configuration parameters e.g., number of mappers, and number of reducers. Then, a polynomial regression is used to model the relation between these configuration parameters and total CPU usage in clock cycles of the job. We also briefly study the influence of input data scaling on measured total CPU usage in clock cycles. This derived model along with the scaling result can then be used to provision the total CPU usage in clock cycles of the same jobs with different input data size. We validate the accuracy of our models using three realistic applications (WordCount, Exim MainLog parsing, and TeraSort). Results show that the predicted total CPU usage in clock cycles of generated resource provisioning options are less than 8% of the measured total CPU usage in clock cycles in our 20-node virtual Hadoop cluster.

show abstract

A study on using uncertain time series matching algorithms for MapReduce applications

Cited by 19 publications

References 21 publications

Big Data Analytics for Dynamic Energy Management in Smart Grids

Big Data Analytics for Dynamic Energy Management in Smart Grids

Process Mining Monitoring for Map Reduce Applications in the Cloud

On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments

Contact Info

Product

Resources

About