‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP

Abstract

A key part of the NLP ethics movement is responsible use of data, but exactly what that means or how it can be best achieved remain unclear. This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them.We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research across the community. Our proposal aims to contribute to the development of a consistent standard for data (re-)use, embraced across NLP conferences.

Resources for Turkish Natural Language Processing: A critical survey

“…Although major computational linguistics venues started to require statements about legal and ethical aspects of data collection and sharing, not all the venues require such statements. It is important to be aware of the existing guidelines, such as ACM code of ethics (Gotterbarn et al, 2018), or the guidelines adapted by major CL conferences, 14 as well as the recent discussion in the field (e.g., Šuster et al, 2017;Rogers et al, 2021). Even though the common guidelines may not fit every task, or every legal jurisdiction, being aware of potential issues, and being explicit about the legal and ethical considerations during data collection and annotation is important.…”

Building Legal Datasets

“…Note that "legal" here refers only to formal law. This distinguishes my scope from (no less important) work on "ethical", "fair" or "responsible" AI [22,13,24]. Despite clear overlaps, neither is a subset of the other.…”

“…To statisticians, better typically means unbiased, though "bias" is used differently from in the bias-variance tradeoff [8], or in algorithmic bias [7]. The growing "responsible AI" literature emphasizes that datasets are better when they are ethically and fairly sourced [22,13,24]. This paper underscores legality as one desideratum for "better".…”

“…The challenge is how we might turn motherhood calls for "multi-disciplinary collaboration" into actionable steps for ML researchers. The rise of ethics guidelines and responsible AI checklists [34,24] offers one solution. In a sense, this involves, ethicists, sociologists, etc pre-computing complex, open-ended obligations into simpler, close-ended compliance heuristics for computer scientists.…”

“…This builds on existing work that already incorporates some legal principles (e.g. [24]) but differs in two ways. First, the framework focuses more on legality and thus complements responsible/ethical AI work.…”