Automatic transparency evaluation for open knowledge extraction systems

Abstract

Background: In this paper a new transparency evaluation framework, Cyrus, is proposed for Open Knowledge Extraction (OKE) systems. Cyrus is based on the state-of-the-art AI and data transparency methods and a well-accepted framework for linked data quality evaluation. Then, the transparency of the outputs of three state-of-the-art OKE systems are automatically evaluated using a combination of three state-of-the-art FAIR (Findable, Accessible, Interoperable, Reusable) tools and the Luzzu linked data quality evaluation framework. OKE systems extract structured knowledge from unstructured or semi-structured text and publish that as linked data. These systems can be used as the fundamental component of advanced knowledge services. However, due to the transition from rule-based to black-box machine learning approaches, most OKE systems are not transparent. This means that their processes and outcomes are not understandable and interpretable for humans. As a result, the transparency problem has been identified as critical by European Union Trustworthy AI guidelines. Automatic transparency evaluation helps with scalability and gives a transparency score that allows comparing different systems' transparencies. It also gives insight into the transparency weaknesses of the system and how to enhance the transparency of the system. Accordingly, being able to evaluate the transparency can be the first step of an effective transparency enhancement process. Results: In Cyrus, data transparency includes 27 dimensions which are grouped in two categories. In this paper, Six of these dimensions, i.e., provenance, interpretability, understandability, licensing, availability, interlinking have been evaluated automatically for three state-of-the-art OKE systems, using the state-of-the-art metrics and tools. The highest mean transparency has been approx. 31\%. Conclusions: This is the first research to study automated transparency evaluation for OKE systems. We show that state-of-the-art FAIR assessment tools and linked data quality evaluation techniques are capable of identifying some transparency weaknesses of OKE systems.\end{abstract}