Towards a comprehensive data lifecycle model for big data environments

Abstract

A huge amount of data is constantly being produced in the world. Data coming from the IoT, from scientific simulations, or from any other field of the eScience, are accumulated over historical data sets and set up the seed for future Big Data processing, with the final goal to generate added value and discover knowledge. In such computing processes, data are the main resource; however, organizing and managing data during their entire life cycle becomes a complex research topic. As part of this, Data LifeCycle (DLC) models have been proposed to efficiently organize large and complex data sets, from creation to consumption, in any field, and any scale, for an effective data usage and big data exploitation.Several DLC frameworks can be found in the literature, each one defined for specific environments and scenarios. However, we realized that there is no global and comprehensive DLC model to be easily adapted to different scientific areas. For this reason, in this paper we describe the Comprehensive Scenario Agnostic Data LifeCycle (COSA-DLC) model, a DLC model which: i) is proved to be comprehensive as it addresses the 6Vs challenges (namely Value, Volume, Variety, Velocity, Variability and Veracity; and ii), it can be easily adapted to any particular scenario and, therefore, fit the requirements of a specific scientific field. In this paper we also include two use cases to illustrate the ease of the adaptation in different scenarios. We conclude that the comprehensive scenario agnostic DLC model provides several advantages, such as facilitating global data management, organization and integration, easing the adaptation to any kind of scenario, guaranteeing good data quality levels and, therefore, saving design time and efforts for the scientific and industrial communities. CCS Concepts• Information systems➝Data management systems • Computer systems organization➝Architectures➝Other architectures➝ Data flow architectures • Software and its engineering➝ Software system structures➝ Data flow architectures

DaLiF: a data lifecycle framework for data-driven governments

“…In GBDE, a data lifecycle model is a key data management tool for organizations [14,15] Data management intends to provide data, which is complete, precise, readable, and accessible to data users. A data lifecycle provides a high-level framework to plan, organize and manage all aspects of data during its life phases, from data planning, collection to its destruction, and the relationship between phases [6,14,14,15,[41][42][43][44][45]. Blazquez, and Domenech highlighted that a data lifecycle is the series of stages that data follow from the moment they enter a system to the moment they are deleted from the system or stored [46].…”

“…There are numerous benefits deriving from designing and implementing in a consistent way a data lifecycle for PAs. These benefits include, but not limited to the followings: (i) ease in planning and handling complexity of data management in all data life phases [15,[42][43][44][53][54][55], (ii) identifying and illustrating a sequence of all essential activities related to data, (iii) support organizations for the preparation of data products for the data users [42-44, 54, 55], (iv) help data users to have a well understanding of the data assets available to them [56], (v) effective gathering of data including metadata from various (internal and external) sources [53,57,58], (vi) implementation of the once-only principle [59], (vii) creation of a homogeneous set of data through consolidation [6,60], (viii) identify, remove noise, uncertainty, and errors in collected data, and maintain data quality [56,61,62], (ix) addition of appropriate data for completion and improvement [61,63], (x) better analysis of data to extract knowledge and discover new insights so that policymakers use this knowledge to generate desire value [42][43][44]61] (xi) visualize data for a better understanding of a common person and its usage for future course of actions [58,64], (xii) support to adopt appropriate data storage approach to ensure the data availability and scalability [15,65], (xiii) assistance to promote the use of data with the consent of the owner of data [66,67], (xiv) create an opportunity to the stakeholders to offer their viewpoints on the data [49,52], (xv) aid PAs to ensure the protection of big data, including personal data, and promote effective governance [6...…”

Methods to Evaluate Lifecycle Models for Research Data Management

“…[5], [6] and [7] are alike to [4] in the approach to review existing models and deriving an own lifecycle model based on a gap analysis. None of the three publications offer generic and empirical evaluation criteria or a metamodel for the existing models.…”

A Novel Architecture for Efficient Fog to Cloud Data Management in Smart Cities

“…The main goals for a DLC model are to operate efficiently, to eliminate waste, and to prepare data products ready for end users matching the expected quality and security constraints. In previous research we proposed the Comprehensive Scenario Agnostic Data LifeCycle (COSA-DLC) model which was proved to be comprehensive, as it was designed as an efficient and global data management model to address the set of 6Vs challenges for big data management (namely Value, Volume, Variety, Velocity, Variability and Veracity), and scenario agnostic, as it is easily adaptable to any scenario or scientific environment [9]. Later, we adapted the COSA-DLC model to a smart city scenario, showing the ease of adaptation of our abstract DLC model.…”