PaaS-BDP - A Multi-Cloud Architectural Pattern for Big Data Processing on a Platform-as-a-Service Model

Vergilio, Thalita; Ramachandran, Muthu

doi:10.5220/0006632400450052

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As containers become more widespread, the risk of vendor lock-in is lowered, since containers enable the decoupling of the processing framework from the infrastructure they run in. Future migration to a safer multi-cloud setup is not only possible, but desirable [64].…”

Section: Cloud Support and Elasticitymentioning

confidence: 99%

“…MC-BDP is an evolution of the PaaS-BDP architectural pattern originally proposed by the authors. While PaaS-BDP introduced a framework-agnostic programming model and enabled different frameworks to share a pool of location and provider-independent resources [64], MC-BDP expands this model by explicitly prescribing a multi-tenant environment where nodes are deployed to multiple clouds. Figure 19 shows a summary of how Facebook, Twitter and Netflix implemented the ten non-functional requirements discussed in this research.…”

Section: Summary and Applicationsmentioning

confidence: 99%

See 1 more Smart Citation

Requirements Engineering for Large-Scale Big Data Applications

Vergilio

Ramachandran

Mullier

2020

Computer Communications and Networks

Self Cite

View full text Add to dashboard Cite

As the use of smart phones proliferates, and human interaction through social media is intensified around the globe, the amount of data available to process is greater than ever before. As consequence, the design and implementation of systems capable of handling such vast amounts of data in acceptable timescales has moved to the forefront of academic and industry-based research. This research represents a unique contribution to the field of Software Engineering for Big Data in the form of an investigation of the big data architectures of three well-known realworld companies: Facebook, Twitter and Netflix. The purpose of this investigation is to gather significant non-functional requirements for real-world big data systems, with an aim to addressing these requirements in the design of our own unique reference architecture for big data processing in the cloud: MC-BDP (Multi-Cloud Big Data Processing). MC-BDP represents an evolution of the PaaS-BDP (Platform as a Service for Big Data Processing) architectural pattern, previously developed by the authors. However, its presentation is not within the scope of this study. The scope of this comparative study is limited to the examination of academic papers, technical blogs, presentations, source code and documentation officially published by the companies under investigation. Ten non-functional requirements are identified and discussed in the context of these companies' architectures: batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault-tolerance, flow control, and flexibility and technology agnosticism. They are followed by the conclusion and considerations for future work.

show abstract

Section: Cloud Support and Elasticitymentioning

confidence: 99%

Section: Summary and Applicationsmentioning

confidence: 99%

Requirements Engineering for Large-Scale Big Data Applications

Vergilio

Ramachandran

Mullier

2020

Computer Communications and Networks

Self Cite

View full text Add to dashboard Cite

show abstract

“…The new reference architecture provided an answer to RQ2, and its publication as a conference paper [78] demonstrated the fulfilment of RO2. A methodology combining post-positivist and interpretivist methods was selected for the evaluation of the proposed reference architecture following a gap in the literature and recent calls for more mixed-methods research in the field of information systems.…”

Section: Discussionmentioning

confidence: 91%

“…MC-BDP is an evolution of the PaaS-BDP architectural pattern originally proposed by the authors. While PaaS-BDP introduced a framework-agnostic programming model for batch and stream processing and enabled different frameworks to share a pool of resources [66], MC-BDP focuses on stream processing and expands the previous model by explicitly prescribing a multi-tenant environment where nodes are deployed to multiple clouds [1]. To reiterate, the rationale for proposing a multi-cloud model is to mitigate the risk of vendor lock-in, perceived as a major obstacle to the adoption of cloud computing.…”

Section: Mc-bdp Reference Architecture For Big Data Stream Processingmentioning

confidence: 99%

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

Vergilio

Kor

Mullier

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The research field of cloud computing has witnessed tremendous progress as commercial cloud providers brought powerful distributed infrastructures within reach of small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the popularisation of containers has empowered virtualisation with seamless orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured and unstructured data emitted at high velocity by an increasing number of internet-enabled devices. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation and big data for a systematic approach to stream processing. The novel contributions of this research are: 1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; 2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings.

show abstract

“…The real-time sampling rate used for the data centre energy efficiency study was, however, limited by the technology available at the time. For the purpose of this research, the data granularity has been enhanced to meet the higher volume and velocity requirements together with MC-BDP's inherent strategy for dealing with late data [60]. Due to operational and availability constraints, a simulation exercise was carried out for the evaluation.…”

Section: Methodsmentioning

confidence: 99%

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

Vergilio,

Kor,

Mullier

2023

Applied Sciences

Self Cite

View full text Add to dashboard Cite

The field of cloud computing has witnessed tremendous progress, with commercial cloud providers offering powerful distributed infrastructures to small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the rise of containers has empowered virtualisation, providing orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured, and unstructured data. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation, and big data for a systematic approach to stream processing. The novel contributions of this research are: (1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; (2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings. The study found that MC-BDP is scalable and fault-tolerant across cloud environments, key attributes for SMEs managing resources under budgetary constraints. Additionally, our experiments on technology agnosticism and container co-location provide new insights into resource utilisation, cost implications, and optimal deployment strategies in cloud-based big data streaming, offering valuable guidelines for practitioners in the field.

show abstract

PaaS-BDP - A Multi-Cloud Architectural Pattern for Big Data Processing on a Platform-as-a-Service Model

Cited by 5 publications

References 0 publications

Requirements Engineering for Large-Scale Big Data Applications

Requirements Engineering for Large-Scale Big Data Applications

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

Contact Info

Product

Resources

About