Cloud based big data infrastructure: Architectural components and automated provisioning

Demchenko, Yuri; Türkmen, Fatih; Laat, Cees de; Blanchet, Christophe; Loomis, Charles

doi:10.1109/hpcsim.2016.7568394

Cited by 11 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This view is useful to understand the interdependencies among collaborators in exchange networks [37,38]. The BDE reveals a complex, connected ecosystem of high-capacity networks, data users, data applications, and services required to filter, store, archive, process, share, and visualize data that is gathered from multiple data sources [39,40].…”

Section: Overview Of the Government Big Data Ecosystem And Data Lifecycle Fieldsmentioning

confidence: 99%

“…The other existing data lifecycles include: Data Lifecycle for HPC Scientific data perspective [114], Data lifecycle for cloud automation tools [40], Data Lifecycle for Telco networks data management [115], Energy big data lifecycle [116], Data lifecycle for the Tobacco industry [117], Data lifecycle for cloud computing [118];Data lifecycle for cloud data [119], Data lifecycle for IoTs [120], Personal data lifecycle [121], Data lifecycle about the coal mine industry [122], Data lifecycle for smart healthcare [123], Data Lifecycle Model for NSF [94], Data lifecycle cycle for smart cities [13], Storage data lifecycle [124], Research data lifecycle [125], lifecycle for CENS Data [126], data lifecycles for industry [127,128], a lifecycle for big scholarly data [129], a lifecycle for social and economic data [46], Data lifecycle for manufacturing [130], Research data lifecycle [131], a lifecycle for big healthcare data [23,132], data lifecycle [133], a lifecycle for environmental research data [134], a lifecycle for big data value creation [26], a lifecycle for big data analytics for psychologists [135], the information pyramid of Reynolds and Busby lifecycle [17], Yuri Demchenko data lifecycle [10], Data lifecycle [58], SCC-data lifecycle [44,54], Data value cycle [136], Lifecycle in databases [137], Knowledge process-lifecycle [138], CMM for Scienti...…”

Section: Rq1: Existing Data Lifecycle Models and Their Phasesmentioning

confidence: 99%

“…• Define filtering criteria to be used by the PAs and researchers to filter the public data, including research data as per their needs [61,62]. • Verification of reliability of the GBD sources as well as of the own data, to manage for any data inconsistencies [40,42] • Responsible for carrying out certain fundamental data transformations to optimize the volume of data flowing from the data collection to the quality phases [42]. • Preparation of additional internal and/or external sources data to be merged with existing public data for data valuation [10,70,143].…”

Section: Collection Phase Key Functions the Data Collection Phase Of Dalif Includes The Following Essential Functionsmentioning

confidence: 99%

See 2 more Smart Citations

DaLiF: a data lifecycle framework for data-driven governments

2021

View full text Add to dashboard Cite

The public sector, private firms, business community, and civil society are generating data that is high in volume, veracity, velocity and comes from a diversity of sources. This kind of data is known as big data. Public Administrations (PAs) pursue big data as “new oil” and implement data-centric policies to transform data into knowledge, to promote good governance, transparency, innovative digital services, and citizens’ engagement in public policy. From the above, the Government Big Data Ecosystem (GBDE) emerges. Managing big data throughout its lifecycle becomes a challenging task for governmental organizations. Despite the vast interest in this ecosystem, appropriate big data management is still a challenge. This study intends to fill the above-mentioned gap by proposing a data lifecycle framework for data-driven governments. Through a Systematic Literature Review, we identified and analysed 76 data lifecycles models to propose a data lifecycle framework for data-driven governments (DaliF). In this way, we contribute to the ongoing discussion around big data management, which attracts researchers’ and practitioners’ interest.

show abstract

Section: Overview Of the Government Big Data Ecosystem And Data Lifecycle Fieldsmentioning

confidence: 99%

Section: Rq1: Existing Data Lifecycle Models and Their Phasesmentioning

confidence: 99%

Section: Collection Phase Key Functions the Data Collection Phase Of Dalif Includes The Following Essential Functionsmentioning

confidence: 99%

See 1 more Smart Citation

DaLiF: a data lifecycle framework for data-driven governments

2021

View full text Add to dashboard Cite

show abstract

“…Kumar et al [31] propose automated provisioning of applications in the cloud using Amazon as an IaaS provider and Ansible as the orchestration engine to automate the deployment. Demchenko et al [32] discuss the importance of automation tools for deployment and management applications for Big Data on the SlipStream cloud automation platform.…”

Section: Other Related Initiatives and Workmentioning

confidence: 99%

Orchestration Architecture for Automatic Deployment of 5G Services from Bare Metal in Mobile Edge Computing Infrastructure

Chirivella‐Perez¹,

Calero²,

Wang³

et al. 2018

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

The progress in realizing the Fifth Generation (5G) mobile networks has been accelerated recently towards deploying 5G prototypes with increasing scale. One of the Key Performance Indicators (KPIs) in 5G deployments is the service deployment time, which should be substantially reduced from the current 90 hours to the target 90 minutes on average as defined by the 5G Public-Private Partnership (5G-PPP). To achieve this challenging KPI, highly automated and coordinated operations are required for the 5G network management. This paper addresses this challenge by designing and prototyping a novel 5G service deployment orchestration architecture that is capable of automating and coordinating a series of complicated operations across physical infrastructure, virtual infrastructure, and service layers over a distributed mobile edge computing paradigm, in an integrated manner. Empirical results demonstrate the superior performance achieved, which meets the 5G-PPP KPI even in the most challenging scenario where 5G services are installed from bare metal.

show abstract

“…The arrival of the big-data technological wave, led to an even higher exploitation of cloud technology 10 . Not only does the sheer amount of data need to be stored, the possibility of at least a primitive form of data analysis and manipulation must be supported.…”

Section: Introductionmentioning

confidence: 99%

Done Yet? A Critical Introspective of the Cloud Management Toolbox

Leznik

Volpert

Griesinger

et al. 2018

2018 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC)

View full text Add to dashboard Cite

With the rapid rise of the cloud computing paradigm, the manual maintenance and provisioning of the technological layers behind it, both in their hardware and virtualized form, became cumbersome and errorprone. This has opened up the need for automated capacity planning strategies in heterogeneous cloud computing environments. However, even with mechanisms to fully accommodate customers and fulfill servicelevel agreements, providers often tend to over-provision their hardware and virtual resources. A proliferation of unused capacity leads to higher energy costs, and correspondingly, the price for cloud technology services. Capacity planning algorithms rely on data collected from the utilized resources. Yet, the amount of data aggregated through the monitoring of hardware and virtual instances does not allow for a manual supervision, much less data analysis or a correlation and anomaly detection. Current data science advancements enable the assistance of efficient automation, scheduling and provisioning of cloud computing resources based on supervised and unsupervised machine learning techniques. In this work, we present the current state of the art in monitoring, storage, analysis and adaptation approaches for the data produced by cloud computing environments, to enable proactive, dynamic resource provisioning.

show abstract

Cloud based big data infrastructure: Architectural components and automated provisioning

Cited by 11 publications

References 12 publications

DaLiF: a data lifecycle framework for data-driven governments

DaLiF: a data lifecycle framework for data-driven governments

Orchestration Architecture for Automatic Deployment of 5G Services from Bare Metal in Mobile Edge Computing Infrastructure

Done Yet? A Critical Introspective of the Cloud Management Toolbox

Contact Info

Product

Resources

About