Location-Aware MapReduce in Virtual Cloud

Geng, Yifeng; Chen, Shimin; Wu, Yongwei; Wu, Ryan; Yang, Guangwen; Zheng, Weimin

doi:10.1109/icpp.2011.40

Cited by 23 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To reduce the contention and interference of disk access and network communication among VMs, Fang et al [20] explored the relationship between I/O scheduling in a virtualization hypervisor and performance of MapReduce applications running on the VMs. Geng et al [21] defined metrics to analyze the data allocation problem in virtual environment theoretically and designed a location-aware file block allocation strategy which retains compatibility with the native hadoop to reduce data transfer between nodes.…”

Section: Related Workmentioning

confidence: 99%

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Jiang

et al. 2014

2014 IEEE 7th International Conference on Cloud Computing

View full text Add to dashboard Cite

With the rapid development of big data and cloud computing, big data analytics as a service in the cloud is becoming increasingly popular. More and more individuals and organizations tend to rent virtual cluster to store and analyze data rather than building their own data centers. However, in virtualization environment, whether scaling out using a cluster with more nodes to process big data is better than scaling up by adding more resources to the original virtual machines (VMs) in cluster is not clear. In this paper, we study the scalability performance issues of hadoop virtual cluster with cost consideration. We first present the design and implementation of VirtualMR platform which can provide users with scalable hadoop virtual cluster services for the MapReduce based big data analytics. Then we run a series of hadoop benchmarks and real parallel machine learning algorithms to evaluate the scalability performance, including scale-up method and scale-out method. Finally, we integrate our platform with resource monitoring module and propose a system tuner. By analyzing the monitored data, we dynamically adjust the parameters of hadoop framework and virtual machine configuration to improve resource utilization and reduce rent cost. Experimental results show that the scaleup method outperforms the scale-out method for CPU-bound applications, and it is opposite for I/O-bound applications. The results also verify the efficiency of system tuner to increase resource utilization and reduce rent cost.

show abstract

Section: Related Workmentioning

confidence: 99%

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Jiang

et al. 2014

2014 IEEE 7th International Conference on Cloud Computing

View full text Add to dashboard Cite

show abstract

“…The map outputs are directly committed to the distributed file system, so there is no Reduce task in RandomWriter. RandomWriter is I/Ointensive [14].…”

Section: Mapreduce Applicationsmentioning

confidence: 99%

“…WordCount performs extensive data reduction on its inputs, so it is CPU-intensive and transfers very little data over the network [17,14].…”

Section: Mapreduce Applicationsmentioning

confidence: 99%

Failure scenario as a service (FSaaS) for Hadoop clusters

Faghri

Bazarbayev

Overholt

et al. 2012

Proceedings of the Workshop on Secure and Dependable Middleware for Cloud Monitoring and Management

View full text Add to dashboard Cite

As the use of cloud computing resources grows in academic research and industry, so does the likelihood of failures that catastrophically affect the applications being run on the cloud. For that reason, cloud service providers as well as cloud applications need to expect failures and shield their services accordingly. We propose a new model called Failure Scenario as a Service (FSaaS). FSaaS will be utilized across the cloud for testing the resilience of cloud applications. In an effort to provide both Hadoop service and application vendors with the means to test their applications against the risk of massive failure, we focus our efforts on the Hadoop platform. We have generated a series of failure scenarios for certain types of jobs. Customers will be able to choose specific scenarios based on their jobs to evaluate their systems.

show abstract

“…Thus, how to reduce I/O tasks becomes a critical challenge in the cloud. Reference [94] allocates data across all VMs based on the data locality to reduce I/O overheads. To relieve the waste of network bandwidth, due to communications between VMs for I/O-intensive applications, [95] proposes a decentralized affinity-aware migration technique that monitors fingerprints of traffic exchanges between VMs and dynamically adjusts/migrates VM placement for communication optimization.…”

Section: Mobile Sensing and Cloud-enabled Decisionsmentioning

confidence: 99%

Future Research on Cyber-Physical Emergency Management Systems

Gelenbe

2013

Future Internet

View full text Add to dashboard Cite

Cyber-physical systems that include human beings and vehicles in a built environment, such as a building or a city, together with sensor networks and decision support systems have attracted much attention. In emergencies, which also include mobile searchers and rescuers, the interactions among civilians and the environment become much more diverse, and the complexity of the emergency response also becomes much greater. This paper surveys current research on sensor-assisted evacuation and rescue systems and discusses the related research issues concerning communication protocols for sensor networks, as well as several other important issues, such as the integrated asynchronous control of large-scale emergency response systems, knowledge discovery for rescue and prototyping platforms. Then, we suggest directions for further research.

show abstract

Location-Aware MapReduce in Virtual Cloud

Cited by 23 publications

References 8 publications

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Failure scenario as a service (FSaaS) for Hadoop clusters

Future Research on Cyber-Physical Emergency Management Systems

Contact Info

Product

Resources

About