Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

Bender, Emily M.; Friedman, Batya

doi:10.1162/tacl_a_00041

Cited by 556 publications

(429 citation statements)

References 38 publications

Supporting

Mentioning

362

Contrasting

Order By: Relevance

“…Finally, the fact sheet discusses the validation process used to evaluate and prove the effectiveness of an explainability approach by describing a user study or synthetic approach that was carried out. Since the Fact Sheet can evolve over time, we encourage their creators to systematically version them [2], thereby making their recipients aware of any updates. Furthermore, indicating whether the whole Fact Sheet or some of its parts are with respect to a (theoretical) algorithmic approach, an actual implementation or a mixture of the two will benefit its clarity.…”

Section: Explainability Fact Sheets Dimensionsmentioning

confidence: 99%

“…Another approach towards clarifying explainability properties in ML is self-reporting and certification. Approaches such as "data statements" [2], "data sheets for data sets" [11] and "nutrition labels for data sets" [16] can help to characterise a data set in a coherent way. Kelley et al [19] argued for a similar concept ("nutrition labels for privacy") to assess privacy of systems that handle personal (and sensitive) information.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Explainability fact sheets

Sokol

Flach

2020

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

179

View full text Add to dashboard Cite

Explanations in Machine Learning come in many forms, but a consensus regarding their desired properties is yet to emerge. In this paper we introduce a taxonomy and a set of descriptors that can be used to characterise and systematically assess explainable systems along five key dimensions: functional, operational, usability, safety and validation. In order to design a comprehensive and representative taxonomy and associated descriptors we surveyed the eXplainable Artificial Intelligence literature, extracting the criteria and desiderata that other authors have proposed or implicitly used in their research. The survey includes papers introducing new explainability algorithms to see what criteria are used to guide their development and how these algorithms are evaluated, as well as papers proposing such criteria from both computer science and social science perspectives. This novel framework allows to systematically compare and contrast explainability approaches, not just to better understand their capabilities but also to identify discrepancies between their theoretical qualities and properties of their implementations. We developed an operationalisation of the framework in the form of Explainability Fact Sheets, which enable researchers and practitioners alike to quickly grasp capabilities and limitations of a particular explainable method. When used as a Work Sheet, our taxonomy can guide the development of new explainability approaches by aiding in their critical evaluation along the five proposed dimensions.

show abstract

Section: Explainability Fact Sheets Dimensionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Explainability fact sheets

Sokol

Flach

2020

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

179

View full text Add to dashboard Cite

show abstract

“…However, no multilingual corpora were found, outlining a potential opportunity in developing resources for the area. Notice that the focus of the area on the English language (particularly, based on prose using in journalistic or encyclopedic texts) and its particular characteristics, although natural from an engineering point of view due to the availability of resources, may induce important bias in the area [70]. The use of multilingual resources, as proposed in this work, may come as a solution to this problem, leading to more robust and linguistically supported methods and applications.…”

Section: Answer To Rq4: What Are the Available Multilingual Open Ie Dmentioning

confidence: 99%

Multilingual Open Information Extraction: Challenges and Opportunities

et al. 2019

View full text Add to dashboard Cite

The number of documents published on the Web in languages other than English grows every year. As a consequence, the need to extract useful information from different languages increases, highlighting the importance of research into Open Information Extraction (OIE) techniques. Different OIE methods have dealt with features from a unique language; however, few approaches tackle multilingual aspects. In those approaches, multilingualism is restricted to processing text in different languages, rather than exploring cross-linguistic resources, which results in low precision due to the use of general rules. Multilingual methods have been applied to numerous problems in Natural Language Processing, achieving satisfactory results and demonstrating that knowledge acquisition for a language can be transferred to other languages to improve the quality of the facts extracted. We argue that a multilingual approach can enhance OIE methods as it is ideal to evaluate and compare OIE systems, and therefore can be applied to the collected facts. In this work, we discuss how the transfer knowledge between languages can increase acquisition from multilingual approaches. We provide a roadmap of the Multilingual Open IE area concerning state of the art studies. Additionally, we evaluate the transfer of knowledge to improve the quality of the facts extracted in each language. Moreover, we discuss the importance of a parallel corpus to evaluate and compare multilingual systems.

show abstract

“…Gebru et al proposed Datasheets for Datasets [2], a standardized reporting schema for data sets in machine learning, including criteria such as the original motivation for data collection, as well as collection procedure, summary of content, and privacy considerations. Similarly, Bender and Friedman proposed Data Statements [7] specifically tailored toward data sets for natural language processing, adding for instance speaker characteristics. Both Datasheets and Data Statements are manually constructed (whereas we envision generating ML consumer labels semi-automatically).…”

Section: Related Workmentioning

confidence: 99%

Towards Generating Consumer Labels for Machine Learning Models

Seifert

Scherzinger

Wiese

2019

2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)

View full text Add to dashboard Cite

Machine learning (ML) based decision making is becoming commonplace. For persons affected by ML-based decisions, a certain level of transparency regarding the properties of the underlying ML model can be fundamental. In this vision paper, we propose to issue consumer labels for trained and published ML models. These labels primarily target machine learning lay persons, such as the operators of an ML system, the executors of decisions, and the decision subjects themselves. Provided that consumer labels comprehensively capture the characteristics of the trained ML model, consumers are enabled to recognize when human intelligence should supersede artificial intelligence. In the long run, we envision a service that generates these consumer labels (semi-)automatically. In this paper, we survey the requirements that an ML system should meet, and correspondingly, the properties that an ML consumer label could capture. We further discuss the feasibility of operationalizing and benchmarking these requirements in the automated generation of ML consumer labels.

show abstract

Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

Cited by 556 publications

References 38 publications

Explainability fact sheets

Explainability fact sheets

Multilingual Open Information Extraction: Challenges and Opportunities

Towards Generating Consumer Labels for Machine Learning Models

Contact Info

Product

Resources

About