More than Text: Multi-modal Chinese Word Segmentation

Abstract

Chinese word segmentation (CWS) is undoubtedly an important basic task in natural language processing. Previous works only focus on the textual modality, but there are often audio and video utterances (such as news broadcast and face-to-face dialogues), where textual, acoustic and visual modalities normally exist. To this end, we attempt to combine the multimodality (mainly the converted text and actual voice information) to perform CWS. In this paper, we annotate a new dataset for CWS containing text and audio. Moreover, we propose a time-dependent multi-modal interactive model based on Transformer framework to integrate multi-modal information for word sequence labeling. The experimental results on three different training sets show the effectiveness of our approach with fusing text and audio.

Mining Word Boundaries in Speech as Naturally Annotated Word Segmentation Data

“…To collect data for the newswire domain, we follow Zhang et al (2021) [9] and adopt the Xuexi platform, 1 on which official news released by our government. We obtain 513 speech-text articles that are read out by human announcers.…”

“…Recently, with the rapid development in speech [7,8] and multi-modal fields [9,10,11], researchers have attempted to introduce speech modality as extra auxiliary information for NLP tasks and shown stable improvements. One mainstream way is to explicitly or implicitly integrate speech modality features into NLP models.…”

“…One mainstream way is to explicitly or implicitly integrate speech modality features into NLP models. For example, Zhang et al (2021) [9] explicitly integrate character-level speech features for CWS task by proposing a time-dependent multi-modal interactive model. Sui et al (2021) [10] introduce a multi-modal multitask model (M3T) to implicitly leverage speech features for Chinese named entity recognition (NER) by training NER and speech-to-text alignment tasks jointly.…”