Big Data-Oriented PaaS Architecture with Disk-as-a-Resource Capability and Container-Based Virtualization

Abstract

With the increasing adoption of Big Data technologies as basic tools for the ongoing Digital Transformation, there is a high demand for data-intensive applications. In order to e ciently execute such applications, it is vital that cloud providers change the way hardware infrastructure resources are managed to improve their performance. However, the increasing use of virtualization technologies to achieve an e cient usage of infrastructure resources continuously widens the gap between applications and the underlying hardware, thus decreasing resource e ciency for the end user. Moreover, this scenario is especially troublesome for Big Data applications, as storage resources are one of the most heavily virtualized, thus imposing a significant overhead for large-scale data processing. This paper proposes a novel PaaS architecture specifically oriented for Big Data where the scheduler o↵ers disks as resources alongside the more common CPU and memory resources, looking forward to provide a better storage solution for the user. Furthermore, virtualization overheads are reduced to the bare minimum by replacing heavy hypervisor-based technologies with operating-system-level virtualization based on light software containers. This architecture has been deployed on a Big Data infrastructure at the CESGA supercomputing center, used as a testbed to compare its performance with OpenStack, a popular private cloud platform. Results have shown significant performance improvements, reducing the execution time of representative Big Data workloads by up to 4.5x.

Provisioning Input and Output Data Rates in Data Processing Frameworks

“…Malensek et al [19] studied a disk I/O contention issue between virtual machines and proposed a scheduler based on Linux tools to provide the share of a disk I/O. Enes et al [13] investigated the resource contention and its effects on PaaS services. They proposed a Platform as a Service (PaaS) architecture where a disk I/O is offered as resources.…”