Data and its (dis)contents: A survey of dataset development and use in machine learning research

Paullada, Amandalynne; Raji, Inioluwa Deborah; Bender, Emily M.; Denton, Emily; Hanna, Alex

doi:10.1016/j.patter.2021.100336

Cited by 325 publications

(181 citation statements)

References 79 publications

Supporting

Mentioning

135

Contrasting

Unclassified

Order By: Relevance

“…In this final section, we also echo other critical work in machine learning (Paullada et al, 2021;Hutchinson et al, 2021) and argue that understanding (speech) datasets as increasingly important infrastructure is useful. It allows us to reframe the task of speech technology development from one primarily done by corporations for markets to one done by a wider range of actors for speech communities.…”

Section: Towards Better Practicessupporting

confidence: 71%

Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR

Markl¹,

McNulty²

2022

Preprint

View full text Add to dashboard Cite

Despite the fact that variation is a fundamental characteristic of natural language, automatic speech recognition systems perform systematically worse on non-standardised and marginalised language varieties. In this paper we use the lens of language policy to analyse how current practices in training and testing ASR systems in industry lead to the data bias giving rise to these systematic error differences. We believe that this is a useful perspective for speech and language technology practitioners to understand the origins and harms of algorithmic bias, and how they can mitigate it. We also propose a re-framing of language resources as (public) infrastructure which should not solely be designed for markets, but for, and with meaningful cooperation of, speech communities.

show abstract

Section: Towards Better Practicessupporting

confidence: 71%

Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR

Markl¹,

McNulty²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Machine learning, computer vision, and social media studies often use "found" data [Hemphill et al 2021;Jo and Gebru 2020;Paullada et al 2021] and render curatorial decisions such as "what data should be available, " "in which format(s) should data be provided, " or "how should this data be sampled" invisible. For instance, datasets scraped from the web (such as Flickr photos [Scheuerman et al 2021;Zhang et al 2015] or Wikipedia talk pages [Wulczyn et al 2016[Wulczyn et al , 2017) suffer from biases in representation [Jo and Gebru 2020].…”

Section: What Renders Data Curation Invisible?mentioning

confidence: 99%

The craft and coordination of data curation: complicating "workflow" views of data science

Thomer¹,

Akmon²,

Jeremy³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data curation is the process of making a dataset fit-for-use and archiveable. It is critical to data-intensive science because it makes complex data pipelines possible, makes studies reproducible, and makes data (re)usable. Yet the complexities of the hands-on, technical and intellectual work of data curation is frequently overlooked or downplayed. Obscuring the work of data curation not only renders the labor and contributions of the data curators invisible; it also makes it harder to tease out the impact curators' work has on the later usability, reliability, and reproducibility of data. To better understand the specific work of data curation -and thereby, explore ways of showing curators' impact -we conducted a close examination of data curation at a large social science data repository, the Inter-university Consortium of Political and Social Research (ICPSR). We asked, What does curatorial work entail at ICPSR, and what work is more or less visible to different stakeholders and in different contexts? And, how is that curatorial work coordinated across the organization? We triangulate accounts of data curation from interviews and records of curation in Jira tickets to develop a rich and detailed account of curatorial work. We find that curators describe a number of craft practices needed to perform their work, which defies the rote sequence of events implied by many lifecycle or workflow models. Further, we show how best practices and craft practices are deeply intertwined. CCS Concepts: • Human-centered computing → Collaborative and social computing systems and tools; • Applied computing → Document preparation; • Information systems → Digital libraries and archives.

show abstract

“…Finally, a common opinion in machine learning [ 96 ] has been that, given enough data and capacity, machine learning bias generally has a vanishing influence over the resulting bias in the learned solution. On the contrary, scale can obfuscate [ 82 ] misspecifications in the task and/or data collection design [ 97 , 98 ]. Here, we focused on how misspecifications in the algorithm design for anomaly detection can result in gross failure even in the ideal theoretical settings of infinite data and capacity.…”

Section: Broader Impactmentioning

confidence: 99%

Perfect Density Models Cannot Guarantee Anomaly Detection

Lan

Dinh

2021

Entropy

View full text Add to dashboard Cite

Thanks to the tractability of their likelihood, several deep generative models show promise for seemingly straightforward but important applications like anomaly detection, uncertainty estimation, and active learning. However, the likelihood values empirically attributed to anomalies conflict with the expectations these proposed applications suggest. In this paper, we take a closer look at the behavior of distribution densities through the lens of reparametrization and show that these quantities carry less meaningful information than previously thought, beyond estimation issues or the curse of dimensionality. We conclude that the use of these likelihoods for anomaly detection relies on strong and implicit hypotheses, and highlight the necessity of explicitly formulating these assumptions for reliable anomaly detection.

show abstract

Data and its (dis)contents: A survey of dataset development and use in machine learning research

Cited by 325 publications

References 79 publications

Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR

Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR

The craft and coordination of data curation: complicating "workflow" views of data science

Perfect Density Models Cannot Guarantee Anomaly Detection

Contact Info

Product

Resources

About