A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

Abstract

The research field of cloud computing has witnessed tremendous progress as commercial cloud providers brought powerful distributed infrastructures within reach of small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the popularisation of containers has empowered virtualisation with seamless orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured and unstructured data emitted at high velocity by an increasing number of internet-enabled devices. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation and big data for a systematic approach to stream processing. The novel contributions of this research are: 1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; 2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings.

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

“…The router uses TCP and UDP connections to transfer packets to other routers in the network, connected via the network bridges. Packets destined for containers located within the same host are handled by the kernel directly without going through the network [82]. In the context of the proposed reference architecture, this feature is particularly desirable since commercial cloud consumers are charged for the number of packets transferred over the network.…”