A reference architecture for big data systems

Sang, Go Muan; Xu, Lai; Vrieze, Paul de

doi:10.1109/skima.2016.7916249

Cited by 19 publications

(21 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The RA presented in this paper can be compared to other published RAs, which have been developed for big data systems. The main difference to the earlier RA by Sang [12,13] is fewer amount of processing layers considered (in [12]), and a bit more fine-grained notation utilised for describing architectural elements (in [12]). Sang included 'Job and Model specification' related elements to the RA on a high level quite similarly as we did [4].…”

Section: Discussionmentioning

confidence: 99%

“…Bolster [16] extended Lambdaarchitecture with a semantic layer/repository, which may be depicted with an Enterprise store in our RA. It can be concluded that none of the published RAs [12][13][14][15][16] include detailed aspects of model development and deployment especially in the edge computing context. Also, we considered deployment environment view of the architecture, and a vision regarding the role of the actors at the business layer.…”

Section: Discussionmentioning

confidence: 99%

“…Then, the identified elements were mapped into the high level architecture view of the RA [4] (see Figs. 6,7,8,9,10,11,12,13,14,15,16,17,18,19 and 20 for detailed mapping figures). Subsequently, the mapped implementation architectures of individual papers were merged into an updated RA, which included existing [4] and new elements ( Fig.…”

Section: Methodsmentioning

confidence: 99%

“…architects, application builders) of RAs had different concerns regarding observed benefits and drawbacks. Sang et al [12] present a RA based on implementation architectures from Facebook, Amazon, Twitter, and Netflix. The RA has four layers (Data collection, processing and loading, Data analysis and aggregation, Interface and visualisation, Job and model specification) into which processing and data stores are mapped to (e.g.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Extending reference architecture of big data systems towards machine learning in edge computing environments

Pääkkönen

Pakkala

2020

J Big Data

View full text Add to dashboard Cite

IntroductionMany big data systems have been developed and realised to provide end user services (Netflix, Facebook, Twitter, LinkedIn etc.). Also, underlying architectures and technologies of the enabling systems have been published [1-3], and RAs have been designed and proposed [4][5][6]. Edge/5G computing is an emerging technological field [7], and the first products are being shipped to the markets. However, the utilisation of machine learning (ML) as part of the edge computing infrastructure is still an area for further research [8]. Particularly, it should be understood, how data is collected, and how models are Abstract Background: Augmented reality, computer vision and other (e.g. network functions, Internet-of-Things (IoT)) use cases can be realised in edge computing environments with machine learning (ML) techniques. For realisation of the use cases, it has to be understood how data is collected, stored, processed, analysed, and visualised in big data systems. In order to provide services with low latency for end users, often utilisation of ML techniques has to be optimized. Also, software/service developers have to understand, how to develop and deploy ML models in edge computing environments. Therefore, architecture design of big data systems to edge computing environments may be challenging. Findings:The contribution of this paper is reference architecture (RA) design of a big data system utilising ML techniques in edge computing environments. An earlier version of the RA has been extended based on 16 realised implementation architectures, which have been developed to edge/distributed computing environments. Also, deployment of architectural elements in different environments is described. Finally, a system view is provided of the software engineering aspects of ML model development and deployment. Conclusions:The presented RA may facilitate concrete architecture design of use cases in edge computing environments. The value of RAs is reduction of development and maintenance costs of systems, reduction of risks, and facilitation of communication between different stakeholders.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Extending reference architecture of big data systems towards machine learning in edge computing environments

Pääkkönen

Pakkala

2020

J Big Data

View full text Add to dashboard Cite

show abstract

“…They try to shed more light on the field of Big Data, present its advantages and inconvenient [56]. The majority focuses on technical challenges, and the latest advances but also on analysis methods and tools [ Various approaches have been used by researchers in order to try to come up with a reference architecture that could be used across industries in a wide variety of use cases [39,40,47,49,50]. Most of those architectures were derived from existing Big Data projects whose components were grouped into modules.…”

Section: Literature Surveymentioning

confidence: 99%

Big Data Architectures: A Detailed and Application Oriented Analysis

KoffiKalipe¹,

Behera²

2019

IJITEE

View full text Add to dashboard Cite

Big Data refers to huge amounts of heterogeneous data from both traditional and new sources, growing at a higher rate than ever. Due to their high heterogeneity, it is a challenge to build systems to centrally process and analyze efficiently such data which are internal and external to organizations. A Big data architecture describes the blueprint of a system handling massive volume of data during its storage, processing, analysis and visualization. Several architectures belonging to different categories have been proposed by academia and industry but the field is still lacking benchmarks. Therefore, a detailed analysis of the characteristics of the existing architectures is required in order to ease the choice between architectures for specific use cases or industry requirements. The types of data sources, the hardware requirements, the maximum tolerable latency, the fitment to industry, the amount of data to be handled are some of the factors that need to be considered carefully before making the choice of an architecture of a Big Data system. However, the wrong choice of architecture can result in huge decline for a company reputation and business. This paper reviews the most prominent existing Big Data architectures, their advantages and shortcomings, their hardware requirements, their open source and proprietary software requirements and some of their realworld use cases catering to each industry. The purpose of this body of work is to equip Big Data architects with the necessary resources to make better informed choices to design optimal Big Data systems.

show abstract