Requirements Engineering for Large-Scale Big Data Applications

Abstract

As the use of smart phones proliferates, and human interaction through social media is intensified around the globe, the amount of data available to process is greater than ever before. As consequence, the design and implementation of systems capable of handling such vast amounts of data in acceptable timescales has moved to the forefront of academic and industry-based research. This research represents a unique contribution to the field of Software Engineering for Big Data in the form of an investigation of the big data architectures of three well-known realworld companies: Facebook, Twitter and Netflix. The purpose of this investigation is to gather significant non-functional requirements for real-world big data systems, with an aim to addressing these requirements in the design of our own unique reference architecture for big data processing in the cloud: MC-BDP (Multi-Cloud Big Data Processing). MC-BDP represents an evolution of the PaaS-BDP (Platform as a Service for Big Data Processing) architectural pattern, previously developed by the authors. However, its presentation is not within the scope of this study. The scope of this comparative study is limited to the examination of academic papers, technical blogs, presentations, source code and documentation officially published by the companies under investigation. Ten non-functional requirements are identified and discussed in the context of these companies' architectures: batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault-tolerance, flow control, and flexibility and technology agnosticism. They are followed by the conclusion and considerations for future work.

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

“…The same conclusion is drawn for three workers [(Beam, Flink) x (single, multi)], and six workers [(Beam, Flink) x (single, multi)]. The means of number of workers (3,6,10) as well as number of containers per node (1,2,4) were also not significantly different from each other for total number of GB sent over the network by windowing rate. Results (in the form graphs and regression models) for the number of gigabytes sent over the network by velocity, windowing rate and cluster are tabulated in Table 10.…”

“…There is a significant difference for three workers (Flink) (single x multi), (Flink) [three workers (single) x 6 workers (single, multi)], while the rest of the combinations show no significant difference. The means of different numbers of workers (3,6,10) are significantly different from each other, while the number of containers per node (1,2,4) are not significantly different from each other for total number of GB received over the network by windowing rate. Increased network utilisation as clusters increased in size was an expected effect, since more workers mean greater parallelisation of work, which optimises performance, but increases container-to-container communication [48].…”

“…They were published as a conference paper and presented at the 13 th International Conference on Software Technologies [29]. An extended version of the paper was subsequently selected for publication as a book chapter on requirements engineering for large-scale big data applications [1]. RO1 was thus fulfilled following peer review of the aforementioned publications.…”

“…To address this gap, our research aims to leverage the latest advances in containers and container orchestration technology for a new PaaS-based multi-cloud reference architecture targeted at big data stream processing. As part of the requirements engineering phase, we have conducted a critical review of published work by three major big data corporations: Facebook, Twitter and Netflix [1], [29]. Ten non-functional requirements were identified and discussed in the context of these companies' architectures: batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault tolerance, flow control, and flexibility and technology agnosticism.…”

Understanding the communication and collaboration challenges encountered by technology managers

“…In this context, technology managers face a number of challenges owing to the constant requirements for upgrade-project rollouts, legacy systems management, ensuring effective knowledge management, and coping with demand and supply imbalance (Agarwal and Ferratt, 2002). As we move further towards big data applications and corresponding data-driven business models (Vergilio et al, 2020), technology managers face additional challenges through attempting to become more data and information competent in order to create value (Vidgen et al, (2017).…”