Hybrid method for modeless Japanese input using N-gram based binary classification and dictionary

Ikegami, Yukino; Tsuruta, Setsuo

doi:10.1007/s11042-013-1805-1

Cited by 3 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of similar tasks involving codeswitched text, the top-performing MAN-EN language identification system achieved an F1 of 0.892 (Chittaranjan et al, 2014), but the annotation scheme includes a category for named entities. Ikegami and Tsuruta (2015) achieved an F1 of 0.97) on codeswitched Japanese-English text input using an n-gram-based approach.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Accurate Pinyin-English Codeswitched Language Identification

Xia¹,

Cheung

2016

Proceedings of the Second Workshop on Computational Approaches to Code Switching

View full text Add to dashboard Cite

Pinyin is the most widely used romanization scheme for Mandarin Chinese. We consider the task of language identification in Pinyin-English codeswitched texts, a task that is significant because of its application to codeswitched text input. We create a codeswitched corpus by extracting and automatically labeling existing Mandarin-English codeswitched corpora. On language identification, we find that SVM produces the best result when using word-level segmentation, achieving 99.3% F1 on a Weibo dataset, while a linear-chain CRF produces the best result at the letter level, achieving 98.2% F1. We then pass the output of our models to a system that converts Pinyin back to Chinese characters to simulate codeswitched text input. Our method achieves the same level of performance as an oracle system that has perfect knowledge of token-level language identity. This result demonstrates that Pinyin identification is not the bottleneck towards developing a Chinese-English codeswitched Input Method Editor, and future work should focus on the Pinyinto-Chinese character conversion step.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Research in improving the codeswitched text input experience also exists for other languages that use a non-alphabetic writing system, such as Japanese. Ikegami and Tsuruta (2015) propose a modeless Japanese input method that automatically switches the input mode using models with n-gram based binary classification and dictionary.…”

Section: Related Workmentioning

confidence: 99%

Accurate Pinyin-English Codeswitched Language Identification

Xia¹,

Cheung

2016

Proceedings of the Second Workshop on Computational Approaches to Code Switching

View full text Add to dashboard Cite

show abstract

“…To improve over the "pure" n-gram approach, we previously proposed a hybrid method [24] that combines n-gram with non-Japanese word dictionary matching. Using the dictionary, false positives against non-Japanese words decrease [12].…”

Section: Related Work a Input Methodsmentioning

confidence: 99%

“…Both n-gram discrimination and non-Japanese word dictionary matching are exploited before performing Kana-to-Kanji conversion. Our previous method [24] follows the line of an open source software, Mozc, derived from the original Google IME. Using n-gram based discrimination jointly with dictionary matching, the Mozc techniques make up for the shortcomings of the other.…”

Section: Related Work a Input Methodsmentioning

confidence: 99%

Word Prediction Model using an elaborated RNN integrated with N-gram for efficient Text Input

Ikegami¹,

Tsuruta

Kutics³

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Smartphone users are beyond two billion worldwide. Heavy users of the texting application rely on input prediction to reduce typing effort. In languages based on the Roman alphabet, many techniques are available. However, Japanese text is based on multiple character sets such as Kanji, Hiragana and Katakana. For its time intensive input, next word prediction an open challenge. To tackle this, a hybrid language model is proposed. It integrates a Recurrent Neural Network (RNN) with an n-gram model. RNNs are powerful models for learning long sequences for next word prediction. N-gram models are best at current word completion. Our RNN language model predicts next words. According the “price” of the performance gain paid by a higher time complexity, our model best deploys on a client-server architecture. Heavily-loaded RNN-LM deploys on the server while the n-gram model on the client. It consists of an input layer equipped with word embedding, an output layer, and hidden layers connected with LSTMs (Long Short-Term Memories). Training is done via BPTT (Back Propagation Through Time). For robust training, BPTT is elaborated by learning rate refinement and gradient norm scaling. To avoid overfitting, the dropout technique is applied except for LSTM. Due to synergetic elaboration, it shows 10% lower perplexity than Zaremba’s excellent conventional models in our experiment. Our model has been incorporated into IME (Input Method Editor) we call Flick. In our experiment, Flick outperforms Mozc (Google Japanese Input) by 16% in time and 34% in the number of key strokes.

show abstract