Efficient Provenance Management via Clustering and Hybrid Storage in Big Data Environments

Hu, Die; Feng, Dan; Xie, Yulai; Xu, Guoying; Gu, Xinrui; Long, Darrell D. E.

doi:10.1109/tbdata.2019.2907116

Cited by 15 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The major issue in centralized big data is addressing security and privacy issues and low scalability since it does not support or tolerate a large number of users. In [33], cluster-based hybrid data storage is implemented for provenance management. In particular, there are three phases used: Clustering, Hybrid Storage and Feedback.…”

Section: Problem Statementmentioning

confidence: 99%

Centralization of Big Data Using Distributed Computing Approach in IoT

Sawiris¹,

El-Gaber²,

Abdel-Fattah³

2021

IJIES

View full text Add to dashboard Cite

Centralized big data is an emerging field that manages information in one common repository. Currently, Internet of Things (IoT) devices are increasing massively for sensing and transmitting data to the big data environment. This information is sensitive and easily traced by attackers due to the centralized data management. To address the security and privacy issues and avoid the system in a single point of failure, in this paper the researchers proposed a new model, namely, BOBS CRABID i.e. Blockchain Based Secure Centralized Big Data model using distributed computing approach. In big data, Hadoop MapReduce is one of the distributed computing approaches which handles all information in a distributed manner. The BOBS CRABID model is designed by three ideas as Multi-Factor authentication, IoT Devices Data Collection and Processing, and Optics based MapReduce for Data Clustering. In multi-factor authentication, a lightweight camellia key generation algorithm (LCKGA) is used for their authenticating all devices based on ID, IP address, MAC and PUF. These credentials are saved in Blockchain Security Entity for authentication verification. All transactions are hashed and stored in the blockchain using Keccak hash algorithm which performs better than the traditional hashing (SHA) 2. Then the researchers conducted IoT devices data collection and pre-processing using Cross Correlation and Min-Max Normalization for redundant data pruning and normalization, respectively. Finally, clustering is performed in a distributed way for optimizing the storage and scalability performance. For that MapReduce is used in which OPTICS algorithm is applied for clustering based on the data size, type of context and nature of data (sensitive or non-sensitive). In this way, data is stored in the big data environment. Experiments were conducted for the proposed BOBS CRABID model and compared with the well-known methods using Hadoop 2.7.2. The proposed BOBS-CRABID model achieves better performance in terms of throughput (increase 1500mbps), response time (reduce 150ms), energy consumption (reduce 35%), attack detection rate (increase 30%), accuracy (increase 20%), computation overhead (300 kb), and time reduce 4s).

show abstract

Section: Problem Statementmentioning

confidence: 99%

Centralization of Big Data Using Distributed Computing Approach in IoT

Sawiris¹,

El-Gaber²,

Abdel-Fattah³

2021

IJIES

View full text Add to dashboard Cite

show abstract

“…It may be defined as the process of detecting the lineage and the derivation of data and data objects [5]. The execution environment of transformation such as the library versions, operating systems, and the nodes responsible for excusing the transformation may also be considered as provenance data [3].…”

Section: Background and Related Work On Big-data Provenancementioning

confidence: 99%

“…Although big data provenance has been increasingly gaining attention, it is still in an early stage of maturity. Majority of the existing research on big data provenance focused on the application of data provenance on big data [3], exploring the challenges of capturing, analyzing, visualizing big data provenance, and identifying future research directions. Further, a number of studies are focused on capturing and modeling provenance, visualizing [20], and mining provenance data [13].…”

Section: Background and Related Work On Big-data Provenancementioning

confidence: 99%

“…This was followed by the provision of an overview of 14 research issues and challenges of provenance within big data in which the author highlighted potential future research directions. Similarly, Glavic [3] highlighted the need for provenance in big data. The author argued that without provenance information, it is impossible for a user to understand the relevance of data, estimate its quality, investigate unexpected or inaccurate results, and define meaningful access control policies.…”

Section: Background and Related Work On Big-data Provenancementioning

confidence: 99%

“…They involve many components provided by various vendors which must be integrated together to develop and operate the system. Therefore, tracing the provenance of an object in big data including the collection of evidences and data from various sources to determine the causes and effects is very difficult [3]. Many of big-data applications rely on Wireless Sensor Networks (WSN) for their common activities to process and transmit data among applications [7].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bio-inspired Think-and-Share Optimization for Big Data Provenance in Wireless Sensor Networks

Alkhalil¹,

Ramadan²,

Ahmad³

2019

IJACSA

View full text Add to dashboard Cite

Big data systems are being increasingly adopted by the enterprises exploiting big data applications to manage datadriven process, practices, and systems in an enterprise wide context. Specifically, big data systems and their underlying applications empower enterprises with analytical decision making (e.g., recommender/decision support systems) to optimize organizational productivity, competitiveness, and growth. Despite these benefits, big data applications face some challenges that include but not limited to security and privacy, authenticity, and reliability of critical data that may result in propagation of false information across systems. Data provenance as an approach and enabling mechanism (to identify the origin, manage the creation, and track the propagation of information etc.) can be a solution to above mentioned challenges for data management in an enterprise context. Data provenance solution(s) can help stakeholders and enterprises to assess the quality of data along with authenticity, reliability, and trust of information on the basis of identity, reproducibility and integrity of data. Considering the wide spread adoption of big data applications and the needs for data provenance, this paper focuses on (i) analyzing state-of-the-art for holistic presentation of provenance in big-data applications (ii) proposing a bioinspired approach with underlying algorithm that exploits human thinking approach to support data provenance in Wireless Sensor Networks (WSNs). The proposed 'Think-and-Share Optimization' (TaSO) algorithms modularizes and automates data provenance in WSNs that are deployed and operated in enterprises. Evaluation of TaSO algorithm demonstrates its efficiency in terms of connectivity, closeness to the sink node, coverage, and execution time. The proposed research contextualizes bio-inspired computation to enable and optimize data provenance in WSNs. Future research aims to exploit machine learning techniques (with underlying algorithms) to automate data provenance for big data systems in networked environments.

show abstract

Workflow provenance in the lifecycle of scientific machine learning

Souza¹,

Azevedo²,

Lourenço³

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Machine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

show abstract

Efficient Provenance Management via Clustering and Hybrid Storage in Big Data Environments

Cited by 15 publications

References 24 publications

Centralization of Big Data Using Distributed Computing Approach in IoT

Centralization of Big Data Using Distributed Computing Approach in IoT

Bio-inspired Think-and-Share Optimization for Big Data Provenance in Wireless Sensor Networks

Workflow provenance in the lifecycle of scientific machine learning

Contact Info

Product

Resources

About