Provenance in Context of Hadoop as a Service (HaaS) - State of the Art and Research Directions

Gupta, Himanshu; Mehta, Sameep; Hans, Sandeep; Chatterjee, Bapi; Lohia, Pranay; Rajmohan, C

doi:10.1109/cloud.2017.91

Cited by 5 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…technology solutions therefore also tend to span all stages. This is particularly true for provenance solutions that target machine-learning systems [91].…”

Section: Auditability In the Data-centric Stagesmentioning

confidence: 99%

See 1 more Smart Citation

Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context

Toreini¹,

Aitken²,

Coopamootoo³

et al. 2020

Preprint

View full text Add to dashboard Cite

Concerns about the societal impact of AI-based services and systems has encouraged governments and other organisations around the world to propose AI policy frameworks to address fairness, accountability, transparency and related topics. To achieve the objectives of these frameworks, the data and software engineers who build machine-learning systems require knowledge about a variety of relevant supporting tools and techniques. In this paper we provide an overview of technologies that support building trustworthy machine learning systems, i.e., systems whose properties justify that people place trust in them. We argue that four categories of system properties are instrumental in achieving the policy objectives, namely fairness, explainability, auditability and safety & security (FEAS). We discuss how these properties need to be considered across all stages of the machine learning life cycle, from data collection through run-time model inference. As a consequence, we survey in this paper the main technologies with respect to all four of the FEAS properties, for data-centric as well as model-centric stages of the machine learning system life cycle. We conclude with an identification of open research problems, with a particular focus on the connection between trustworthy machine learning technologies and their implications for individuals and society.

show abstract

“…technology solutions therefore also tend to span all stages. This is particularly true for provenance solutions that target machine-learning systems [91].…”

Section: Auditability In the Data-centric Stagesmentioning

confidence: 99%

“…A recent "roadmap" for the role of provenance in Spark-based scientific workflows is available from Guedes et al [89]. Still in the context of map-reduce based systems, [91] presents state of the art and research directions for provenance in the context of "Hadoop as a Service".…”

Section: Auditability In the Model-centric Stagesmentioning

confidence: 99%

Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context

Toreini¹,

Aitken²,

Coopamootoo³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Much of the literature around auditability relates to data provenance research in database and cloud concepts [44] and model provenance approaches [39]. It involves proposals on how to store provenance data [18,20,21,74,94], summarisation of provenance data [2,28,66], specific query language for the provenance data [28], query explainability [8,10], Natural language processing for provenance data [27,90] and cryptographic solutions to verify model's behaviours without exposing user privacy [72] or revealing model's intellectual properties [87].…”

Section: Auditability Technologiesmentioning

confidence: 99%

The relationship between trust in AI and trustworthy machine learning technologies

Toreini¹,

Aitken²,

Coopamootoo³

et al. 2019

Preprint

View full text Add to dashboard Cite

To build AI-based systems that users and the public can justifiably trust one needs to understand how machine learning technologies impact trust put in these services. To guide technology developments, this paper provides a systematic approach to relate social science concepts of trust with the technologies used in AI-based services and products. We conceive trust as discussed in the ABI (Ability, Benevolence, Integrity) framework and use a recently proposed mapping of ABI on qualities of technologies. We consider four categories of machine learning technologies, namely these for Fairness, Explainability, Auditability and Safety (FEAS) and discuss if and how these possess the required qualities. Trust can be impacted throughout the life cycle of AI-based systems, and we introduce the concept of Chain of Trust to discuss technological needs for trust in different stages of the life cycle. FEAS has obvious relations with known frameworks and therefore we relate FEAS to a variety of international Principled AI policy and technology frameworks that have emerged in recent years.

show abstract

“…Where there is confidential information, there lies the threat as well. Analogous to this, several digital security threats have evolved drastically in the past years [11], [12], [13].…”

Section: Introductionmentioning

confidence: 99%

BTLA-LSDG: Blockchain-Based Triune Layered Architecture for Authenticated Subgraph Query Search in Large-Scale Dynamic Graphs

Sharmila¹,

Devi²

2023

IETE Journal of Research

View full text Add to dashboard Cite

Over the past era, subgraph mining from a large collection of graph database is a crucial problem. Existing works on subgraph mining is based on the threshold value, which returns similar graphs for a query graph. However, the number of graphs in the answer set for the same threshold can vary in terms of structure and context. In addition, scalability is another big problem due to insufficient storage.Further, it can suffer from security issues. In a distributed environment, since more queries come from different users, it is possible for the attackers to access the graph. These three problems highly exist in the current subgraph mining. To address this downside, our proposed work introduces aBlockchain-based Triune Layered Architecture for authenticated query search in the large scale dynamic graphs. The two fold process is handled in BTLA-LSDG: graph indexing and authenticated query search (query processing), which are implemented in triune layers (Data Generation Layer, Data Storage Layer, and Service Layer). Initially, data owners are authenticated to blockchain using Four-Q-Curve algorithm. The graph index is constructed by data owners and merged graph index is constructed by service providers. Based on the uploaded graph index, hash index is constructed using SHA-3. The hash index is fed into blockchain and form Dendrimer -Fractal Index. On the other hand, data user submits query with authentication. For every authenticated query, the four fold process is handled. Firstly, Multi-Constraint based Belief Entropy function is used for feature sets computation for a given query. Then Dual Similarity based MapReduce helps inmapping and reducing the relevant subgraphs with the use of optimal feature sets. Thirdly, Recurrent Neural Network (RNN)is used for subgraph isomorphic testing. Finally, graph index refinement process is undertaken to improve the query results. For this, query error is computed for each user's query at the end of retrieval. With respect to query error, fuzzy logic is used to refine the index of graph dynamically. This experiment is implemented using Hadoop environment and the results show better efficiency in terms of Scalability, Security and Storage. Further, it is tested with Precision, Recall, F-measure, Accuracy, Error Rate, Query Response Time and Positive Results.

show abstract

Provenance in Context of Hadoop as a Service (HaaS) - State of the Art and Research Directions

Cited by 5 publications

References 18 publications

Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context

Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context

The relationship between trust in AI and trustworthy machine learning technologies

BTLA-LSDG: Blockchain-Based Triune Layered Architecture for Authenticated Subgraph Query Search in Large-Scale Dynamic Graphs

Contact Info

Product

Resources

About