A Large-Scale Chinese Multimodal NER Dataset with Speech Clues

Abstract

In this paper, we aim to explore an uncharted territory, which is Chinese multimodal named entity recognition (NER) with both textual and acoustic contents. To achieve this, we construct a large-scale human-annotated Chinese multimodal NER dataset, named CNERTA. Our corpus totally contains 42,987 annotated sentences accompanying by 71 hours of speech data. Based on this dataset, we propose a family of strong and representative baseline models, which can leverage textual features or multimodal features. Upon these baselines, to capture the natural monotonic alignment between the textual modality and the acoustic modality, we further propose a simple multimodal multitask model by introducing a speech-to-text alignment auxiliary task.Through extensive experiments, we observe that: (1) Progressive performance boosts as we move from unimodal to multimodal, verifying the necessity of integrating speech clues into Chinese NER. (2) Our proposed model yields state-of-the-art (SoTA) results on CNERTA, demonstrating its effectiveness. For further research, the annotated dataset is publicly available at http://github.com/DianboWork/ CNERTA.

Multimodal Data Fusion for Few-shot Named Entity Recognition Method

“…Since people usually pause between words when speaking, the speech modality is used as an auxiliary modality to help the model identify the boundaries of entities in the text. Sui et al [24] used Mel filter bank features down-sampled by Convolutional Neural Network (CNN) as speech feature representation and fused with text representation obtained by BERT for entity recognition. (2) Text + font structure.…”

MIntRec: A New Dataset for Multimodal Intent Recognition

“…Some multimodal datasets also detect unique properties of human languages, such as sense of humor [17,18], metaphor [60], sarcasm [7,9]. Moreover, multimodal datasets are designed for a series of other tasks in NLP, such as dialogue act classification [39,40], named entity recognition [43], comprehension and reasoning [49,53], comments generation [48], fake news detection [33], etc. Nevertheless, there is a lack of multimodal datasets for intent analysis in real-world dialogue scenes.…”

A TL_FLAT Model for Chinese Text Datasets of UAV Power Systems: Optimization and Performance

“…However, entity recognition in the feld of aviation manufacturing and assembly mainly concerns the recognition of key features such as the algorithm, parts, parameters, materials, functions, structures, and features involved in web pages, documents, patents, technical reports, etc., which are apparently domestic and foreign. Sui et al [13] proposed a multimodal multitasking algorithm based on their own labeled dataset to explore a multimodal named entity recognition (NER) approach for Chinese textual and auditory content by introducing a speech-to-text alignment assistance task. Zhang et al [14] proposed a machine reading comprehension framework that integrates adaptive positive untagging techniques into NER and experimentally demonstrated that the framework is efective for datasets containing a large number of untagged entities.…”