Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix

Vergilio, Thalita; Ramachandran, Muthu

doi:10.5220/0006825408330840

Cited by 4 publications

(15 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, as new data are continuously added via the data stream that gets collected daily, the model makes use of the speed layer and batch layer of big data technologies for data processing. Finally, the output will be used for reinforcement learning in the servicing layer of the big data architecture that caters to both functional and non-functional requirements [39].…”

Section: Use Case Of Proposed Model For Smart Home Automationmentioning

confidence: 99%

Big Data and Personalisation for Non-Intrusive Smart Home Automation

Asaithambi

Venkatraman²,

Venkatraman

2021

BDCC

View full text Add to dashboard Cite

With the advent of the Internet of Things (IoT), many different smart home technologies are commercially available. However, the adoption of such technologies is slow as many of them are not cost-effective and focus on specific functions such as energy efficiency. Recently, IoT devices and sensors have been designed to enhance the quality of personal life by having the capability to generate continuous data streams that can be used to monitor and make inferences by the user. While smart home devices connect to the home Wi-Fi network, there are still compatibility issues between devices from different manufacturers. Smart devices get even smarter when they can communicate with and control each other. The information collected by one device can be shared with others for achieving an enhanced automation of their operations. This paper proposes a non-intrusive approach of integrating and collecting data from open standard IoT devices for personalised smart home automation using big data analytics and machine learning. We demonstrate the implementation of our proposed novel technology instantiation approach for achieving non-intrusive IoT based big data analytics with a use case of a smart home environment. We employ open-source frameworks such as Apache Spark, Apache NiFi and FB-Prophet along with popular vendor tech-stacks such as Azure and DataBricks.

show abstract

Section: Use Case Of Proposed Model For Smart Home Automationmentioning

confidence: 99%

Big Data and Personalisation for Non-Intrusive Smart Home Automation

Asaithambi

Venkatraman²,

Venkatraman

2021

BDCC

View full text Add to dashboard Cite

show abstract

“…Facebook, Twitter and Netflix) with the purpose of understanding nonfunctional requirements for real-world big data systems. The selection criteria for these companies, methodology and results of this evaluation are explained in a separate publication [29]; RO2 -based on the findings in RO1, propose a new reference architecture based on industry's best practices and focused on vendor lock-in mitigation; RO3 -develop a prototype, based on a case study, to enable the empirical evaluation of the proposed reference architecture; RO4: to identify and implement a set of relevant performance metrics for the prototype in RO3; RO5 -to design and execute distinct sets of experiments to evaluate the proposed reference architecture in terms of the non-functional requirements implemented.…”

Section: Aim and Objectivesmentioning

confidence: 99%

“…Finally, the evaluation of Maier's reference architecture was performed by analysing the real-world big data implementations of Facebook, LinkedIn and Oracle and retrofitting them to the reference architecture proposed [50]. While this research did perform a similar investigation of real-world big data implementations [29], its purpose was not to evaluate the MC-BDP proposal, but to gather requirements for it. The evaluation of MC-BDP was performed through a concrete case-study, with both quantitative and qualitative data collected and thoroughly analysed as part of a rigorous scientific process described in Section 3.…”

Section: Related Literature Reviewmentioning

confidence: 99%

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

Vergilio

Kor

Mullier

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The research field of cloud computing has witnessed tremendous progress as commercial cloud providers brought powerful distributed infrastructures within reach of small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the popularisation of containers has empowered virtualisation with seamless orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured and unstructured data emitted at high velocity by an increasing number of internet-enabled devices. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation and big data for a systematic approach to stream processing. The novel contributions of this research are: 1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; 2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings.

show abstract

“…MOA provides a microservice-oriented modelling approach for designing and implementing the interaction and communication among software service layers to support information sharing among different stakeholders for providing smart services in an ITS. We identify the key software service requirements of our proposed microservice-oriented modelling for smart transportation and analytics [37,38].…”

Section: Service-driven Modelling For Itssmentioning

confidence: 99%

“…Typically, ITSs should be capable of scaling horizontally and handling large amount of traffic coming from several heterogeneous data sources such as IoT devices, smartphones, cameras, machine logs, and social media [38,45]. There are ten non-functional microservice performance requirements in the context of designing contemporary data architectures for intelligent systems: Batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault-tolerance, flow control, and flexibility and technology agnosticism [38]. Conventional big data systems that rely on MapReduce jobs and batch ETL cannot be used because of high latency in ingesting new data [46].…”

Section: Non-functional Microservice Performance Requirements Of Itssmentioning

confidence: 99%

MOBDA: Microservice-Oriented Big Data Architecture for Smart City Transport Systems

Asaithambi

Venkatraman

Venkatraman³

2020

BDCC

View full text Add to dashboard Cite

Highly populated cities depend highly on intelligent transportation systems (ITSs) for reliable and efficient resource utilization and traffic management. Current transportation systems struggle to meet different stakeholder expectations while trying their best to optimize resources in providing various transport services. This paper proposes a Microservice-Oriented Big Data Architecture (MOBDA) incorporating data processing techniques, such as predictive modelling for achieving smart transportation and analytics microservices required towards smart cities of the future. We postulate key transportation metrics applied on various sources of transportation data to serve this objective. A novel hybrid architecture is proposed to combine stream processing and batch processing of big data for a smart computation of microservice-oriented transportation metrics that can serve the different needs of stakeholders. Development of such an architecture for smart transportation and analytics will improve the predictability of transport supply for transport providers and transport authority as well as enhance consumer satisfaction during peak periods.

show abstract

Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix

Cited by 4 publications

References 9 publications

Big Data and Personalisation for Non-Intrusive Smart Home Automation

Big Data and Personalisation for Non-Intrusive Smart Home Automation

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

MOBDA: Microservice-Oriented Big Data Architecture for Smart City Transport Systems

Contact Info

Product

Resources

About