Distributed Storage and Querying Techniques for a Semantic Web of Scientific Workflow Provenance

Abraham, John; Brazier, Pearl; Chebotko, Artem; Casas, Jaime Navarro; Piazza, Anthony

doi:10.1109/scc.2010.14

Cited by 19 publications

(20 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PASS has explored the use of clouds [53,54]. Provbase [1] uses Hbase, an open-source implementation of Google's BigTable [9], to store and query scientific workflow provenance. IBM researchers have proposed a provenance index that improves the execution of forward and backward provenance queries [32].…”

Section: Related Workmentioning

confidence: 99%

SPADE: Support for Provenance Auditing in Distributed Environments

Gehani

Tariq

2012

Lecture Notes in Computer Science

177

138

View full text Add to dashboard Cite

Abstract. SPADE is an open source software infrastructure for data provenance collection and management. The underlying data model used throughout the system is graph-based, consisting of vertices and directed edges that are modeled after the node and relationship types described in the Open Provenance Model. The system has been designed to decouple the collection, storage, and querying of provenance metadata. At its core is a novel provenance kernel that mediates between the producers and consumers of provenance information, and handles the persistent storage of records. It operates as a service, peering with remote instances to enable distributed provenance queries. The provenance kernel on each host handles the buffering, filtering, and multiplexing of incoming metadata from multiple sources, including the operating system, applications, and manual curation. Provenance elements can be located locally with queries that use wildcard, fuzzy, proximity, range, and Boolean operators. Ancestor and descendant queries are transparently propagated across hosts until a terminating expression is satisfied, while distributed path queries are accelerated with provenance sketches.

show abstract

Section: Related Workmentioning

confidence: 99%

SPADE: Support for Provenance Auditing in Distributed Environments

Gehani

Tariq

2012

Lecture Notes in Computer Science

177

138

View full text Add to dashboard Cite

show abstract

“…PASS has explored the use of clouds [24]. Provbase uses Hbase to store and query scientific workflow provenance [27]. Further compressing provenance [26], indexing [14] and optimization techniques [28] have also been considered.…”

Section: Related Workmentioning

confidence: 99%

Distributed data provenance for large-scale data-intensive computing

Zhao

Shou

Malik

et al. 2013

2013 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Abstract-It has become increasingly important to capture and understand the origins and derivation of data (its provenance). A key issue in evaluating the feasibility of data provenance is its performance, overheads, and scalability. In this paper, we explore the feasibility of a general metadata storage and management layer for parallel file systems, in which metadata includes both file operations and provenance metadata. We experimentally investigate the design optimality-whether provenance metadata should be loosely-coupled or tightly integrated with a file metadata storage systems. We consider two systems that have applied similar distributed concepts to metadata management, but focusing singularly on kind of metadata: (i) FusionFS, which implements a distributed file metadata management based on distributed hash tables, and (ii) SPADE, which uses a graph database to store audited provenance data and provides distributed module for querying provenance. Our results on a 32-node cluster show that FusionFS+SPADE is a promising prototype with negligible provenance overhead and has promise to scale to petascale and beyond. Furthermore, FusionFS with its own storage layer for provenance capture is able to scale up to 1K nodes on BlueGene/P supercomputer.

show abstract

“…Several implementations of this theoretical concept exist in the research community. These efforts include the work done by Choi et al [47] and Abraham et al [48]. A separate technique that has been used to store and retrieve RDF data makes use of peer-to-peer systems [49][50][51][52].…”

Section: Overall Related Researchmentioning

confidence: 99%

Cloud-Centric Assured Information Sharing

Thuraisingham

Khadilkar

Rachapalli

et al. 2012

Intelligence and Security Informatics

View full text Add to dashboard Cite

Abstract. In this paper we describe the design and implementation of cloudbased assured information sharing systems. In particular, we will describe our current implementation of a centralized cloud-based assured information sharing system and the design of a decentralized hybrid cloud-based assured information sharing system of the future. Our goal is for coalition organizations to share information stored in multiple clouds and enforce appropriate policies. IntroductionThe advent of cloud computing and the continuing movement toward software as a service (SaaS) paradigms has posed an increasing need for assured information sharing (AIS) as a service in the cloud. The urgency of this need has been voiced as recently as April 2011 by NSA CIO Lonny Anderson in describing the agency's focus on a "cloud-centric" approach to information sharing with other agencies [1]. Likewise, the DoD has been embracing cloud computing paradigms to more efficiently, economically, flexibly, and scalably meet its vision of "delivering the power of information to ensure mission success through an agile enterprise with freedom of maneuverability across the information environment" [2][3][4][5]. Both agencies therefore have a tremendous need for effective AIS technologies and tools for cloud environments. Although a number of AIS tools have been developed over the past five years for policy-based information sharing [5][6][7][8], to our knowledge none of these tools operate in the cloud and hence do not provide the scalability needed to support large numbers of users utilizing massive amounts of data. Recent prototype systems for supporting cloud-based AIS have applied cloud-centric engines that query large amounts of data in relational databases via non-cloud policy engines that enforce policies expressed in XACML [9][10]. While this is a significant improvement over prior efforts (and has given us insights into implementing cloud-based solutions), it nevertheless has at least three significant limitations. First, XACML-based policy specifications are not expressive enough to support many of the complex policies needed for AIS missions like those of the NSA and DoD. Second, to meet the scalability and efficiency requirements of mission-critical tasks, the policy engine needs to operate in the cloud rather than externally. Third, secure query processing based on relational technology has limitations in representing and processing unstructured data needed for command and control applications.

show abstract

Distributed Storage and Querying Techniques for a Semantic Web of Scientific Workflow Provenance

Cited by 19 publications

References 23 publications

SPADE: Support for Provenance Auditing in Distributed Environments

SPADE: Support for Provenance Auditing in Distributed Environments

Distributed data provenance for large-scale data-intensive computing

Cloud-Centric Assured Information Sharing

Contact Info

Product

Resources

About