A framework for fostering transparency in shared artificial intelligence models by increasing visibility of contributions

Abstract

Increased adoption of artificial intelligence (AI) systems into scientific workflows will result in an increasing technical debt as the distance between the data scientists and engineers who develop AI system components and scientists, researchers and other users grows. This could quickly become problematic, particularly where guidance or regulations change and once-acceptable best practice becomes outdated, or where data sources are later discredited as biased or inaccurate. This paper presents a novel method for deriving a quantifiable metric capable of ranking the overall transparency of the process pipelines used to generate AI systems, such that users, auditors and other stakeholders can gain confidence that they will be able to validate and trust the data sources and contributors in the AI systems that they rely on. The methodology for calculating the metric, and the type of criteria that could be used to make judgements on the visibility of contributions to systems are evaluated through models published at ModelHub and PyTorch Hub, popular archives for sharing science resources, and is found to be helpful in driving consideration of the contributions made to generating AI systems and approaches toward effective documentation and improving transparency in machine learning assets shared within scientific communities. K E Y W O R D Saccountability, data ecosystems, data provenance, ML model evaluation, model zoo, transparency INTRODUCTIONScientists and researchers across all fields are making use of artificial intelligence (AI) systems and machine learning (ML) models in their experimentation. Increasingly, these new research assets are created by domain experts curating and aggregating data from multiple and diverse sources; their outputs can be new data assets or heavily data-influenced products, including ML models, which go on to be used in research within the originating organization, or in the wider community when they are published or distributed and shared via gateways with collaborating or even unknown third party research groups and organizations. This is already a real situation-in the UK financial services sector, for example, 24% of ML use cases are developed and implemented by third-party providers, with many of those developing internally also reporting adaptation or further development of off-the-shelf ML models or libraries. 1Providing support to offer transparency and traceability of assets through the production pipeline is an important contributor to delivering accountability, which is necessary to achieve and retain confidence and trust, such that scientists and practitioners using AI systems, whether developed in-house or sourced externally from their community, are able to demonstrate the provenance and authenticity of the data and knowledge they use to make decisions. 2,3 In considering the development workflows for ML models, the contributing assets can be identified and itemized, and will typically include data sources and labeled datasets used for model training and validation, alon...

Users are the North Star for AI Transparency

“…One key thrust for transparency work centers on the inputs required to produce an AI system. These studies focus on the intent behind [Gebru et al, 2021], composition of [Weissgerber et al, 2016], or use limitations for [Bertino et al, 2019a] datasets as well as address the conflicts that can arise between transparency and user concerns about data privacy and security [Jordan et al, 2020;Beam et al, 2020]. Research toward these factors often explores ways to strike a balance between increasing overall transparency to reap the attendant benefits regarding fairness, accountability, and trust, while mitigating the potential losses vis-a-vis privacy and security.…”

“…We consider here any work toward elucidating the functionality and quality of AI systems-including methods directed at both practitioners and users. Practitioners often need to debug models or reproduce results more easily [Beam et al, 2020]. On the other hand, users tend to simply desire a basic overview of a system's function for confidence in its functionality [Mei et al, 2022a] ( §3.1).…”

Automatic transparency evaluation for open knowledge extraction systems

“…• Results from any model updating [13] [ 11] Such as min, max, and median values at the top-10 and over-all [12] Such as recalibration, recalibration, predictor effects adjusted, or new predictors added [13] i.e., model specification, model performance 7. Model evaluation [15,42,44,49,64] • Evaluation Dataset/s [42,44,67] -Test and holdout data transparency information [44] -Dataset size information: Sample size [44,63,64], Rationale for the sample size [63] -Preprocessing techniques used [42] • Comparison between validation and development datasets [63,64] • Methods used to evaluate model performance, e.g., cross validation [64] • Performance measures results [15,42,43,44,49,63,64,69] • Rationale for performance measures [44] • Benchmarking against standard datasets [44,49] • Reliability analysis, e.g., baseline survival [64] • FAIR [57] • Third party performance verifications [44] • Concept drift [44] • Interpretation of results [63,64] -If objectives are met considering the results…”

Editorial: Science Gateways Special Issue 2020

“…Barclay et al 7 address a variety of new problems related to the use of AI methods in scientific workflows. Investigated problems include guidance, regulations change, outdated best‐practices, discredited as biased or inaccurate data sources and similar.…”