‘Beach’ to ‘Bitch’: Inadvertent Unsafe Transcription of Kids’ Content on YouTube

Abstract

Over the last few years, YouTube Kids has emerged as one of the highly competitive alternatives to television for children's entertainment. Consequently, YouTube Kids' content should receive an additional level of scrutiny to ensure children's safety. While research on detecting offensive or inappropriate content for kids is gaining momentum, little or no current work exists that investigates to what extent AI applications can (accidentally) introduce content that is inappropriate for kids. In this paper, we present a novel (and troubling) finding that well-known automatic speech recognition (ASR) systems may produce text content highly inappropriate for kids while transcribing YouTube Kids' videos. We dub this phenomenon as inappropriate content hallucination. Our analyses suggest that such hallucinations are far from occasional, and the ASR systems often produce them with high confidence. We release a first-of-its-kind data set of audios for which the existing state-of-the-art ASR systems hallucinate inappropriate content for kids. In addition, we demonstrate that some of these errors can be fixed using language models.

Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)

“…Grounding Other Research Efforts: Apart from aiding content moderation, inappropriate lexicons can lend grounding to other research efforts. For example, a recent paper (Ramesh et al, 2022) has consulted the CMU Lexicon and another lexicon listing taboo-words for kids (Jay, 1992) to construct a set of inappropriate words for kids. Ramesh et al (2022) take a rather passive stance in their treatment of queer-related words.…”

“…For example, a recent paper (Ramesh et al, 2022) has consulted the CMU Lexicon and another lexicon listing taboo-words for kids (Jay, 1992) to construct a set of inappropriate words for kids. Ramesh et al (2022) take a rather passive stance in their treatment of queer-related words. Ramesh et al (2022) state that the authors extensively debated whether non-pejorative queer-related words such as gay or queer should be in the lexicon, but since these words were already present in both lexicons, they retain them, seeking more inputs from developmental psychologists.…”

“…Ramesh et al (2022) take a rather passive stance in their treatment of queer-related words. Ramesh et al (2022) state that the authors extensively debated whether non-pejorative queer-related words such as gay or queer should be in the lexicon, but since these words were already present in both lexicons, they retain them, seeking more inputs from developmental psychologists. Unless the research community takes a more definitive stance on when and how non-pejorative queer-related words should be included in these inappropriate language lexicons, we may see more research efforts sidestepping this important issue.…”

The Era of Artificial Intelligence Deception: Unraveling the Complexities of False Realities and Emerging Threats of Misinformation

“…Furthermore, the AI community advocates for transparency in AI models, where the reasoning behind AI-generated content is made clear, and the sources of information are verifiable. This approach aims to build trust in AI systems by allowing users to understand and verify the basis of AI outputs, thereby reducing the risk of accepting hallucinated content as truth [6]. The widespread implications of AI hallucinations, as observed on platforms such as YouTube, where AI may inadvertently transcribe unsafe content, underline the need for vigilance in the downstream applications of AI.…”

“…Through concerted efforts to enhance the robustness of AI systems and promote transparency, the AI community can work toward mitigating the risks associated with AI hallucinations and safeguarding the integrity of AI applications. Collective insights from researchers such as [3][4][5][6][7][8][9][10][11] underscore the multifaceted nature of this issue and the diverse strategies required to address it. The potential for AI to inadvertently transcribe unsafe content on platforms such as YouTube is a stark reminder of the challenges posed by AI hallucinations.…”