Optimizing Word Segmentation for Downstream Task

Abstract

In traditional NLP, we tokenize a given sentence as a preprocessing, and thus the tokenization is unrelated to a target downstream task. To address this issue, we propose a novel method to explore a tokenization which is appropriate for the downstream task. Our proposed method, optimizing tokenization (Op-Tok), is trained to assign a high probability to such appropriate tokenization based on the downstream task loss. OpTok can be used for any downstream task which uses a vector representation of a sentence such as text classification. Experimental results demonstrate that OpTok improves the performance of sentiment analysis and textual entailment. In addition, we introduce OpTok into BERT, the state-ofthe-art contextualized embeddings and report a positive effect.

Joint Optimization of Tokenization and Downstream Model

“…We focus on SentencePiece (SP) (Kudo and Richardson, 2018) and OpTok (Hiraoka et al, 2020) as other tokenizers for comparison with the proposed method. OpTok is a method to optimize tokenization for text classification by weighting a sentence vector with N -best tokenization.…”

“…Much of prior research has reported that an appropriate tokenization depends on each downstream task (Xu et al, 2008;Chang et al, 2008;Nguyen et al, 2010;Domingo et al, 2018;Hiraoka et al, 2019;Gowda and May, 2020). Moreover, Hiraoka et al (2020) implies that we have to consider a downstream model to determine an appropriate tokenization. In other words, we can improve the performance of a downstream model by determining an appropriate tokenization for the downstream model.…”

“…Several studies have addressed the optimization of a tokenizer based on a downstream task or/and model (He et al, 2020;Hiraoka et al, 2020), but existing methods are restricted to specific tasks. He et al (2020) proposed DPE as a tokenization method for a sequence-to-sequence problem such as machine translation.…”

“…Their method trains a tokenizer with a given training corpus, but it is isolated from a downstream model such as a neural encoder-decoder for machine translation. Hiraoka et al (2020) proposed OpTok, which jointly trains a tokenizer and a downstream model. However, its architecture is specific to classification problems based on sentence representations, and thus, it cannot be applied for various tasks such as sequence-tosequence problems.…”

Joint Optimization of Tokenization and Downstream Model

“…Much of prior research has reported that an appropriate tokenization depends on each downstream task (Xu et al, 2008;Chang et al, 2008;Nguyen et al, 2010;Domingo et al, 2018;Hiraoka et al, 2019;Gowda and May, 2020). Moreover, Hiraoka et al (2020) implies that we have to consider a downstream model to determine an appropriate tokenization. In other words, we can improve the performance of a downstream model by determining an appropriate tokenization for the downstream model.…”