Improving Efficiency of Sentence Boundary Detection by Feature Selection

Ho, Thi-Nga; Chong, Tze Yuang; Hai, Van; Pham, Van Tung; Chng, Eng Siong

doi:10.1007/978-3-662-49390-8_58

Cited by 3 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, researchers need to understand a language well to apply rule-based approaches. In the past several years, DL models have been successfully applied to many sequential labeling and classification tasks, such as SBD, speech recognition, WS, and POS tagging and chunking [4,5] . DL models can learn a hierarchy of nonlinear feature detectors to capture complex statistical patterns [6] .…”

Section: Introductionmentioning

confidence: 99%

“…Modern Tibetan is composed of letters, including 30 consonant letters and four vowel letters, and consonant letters and vowel letters are arranged separately in the alphabet. The 30 consonant letters are divided into eight groups, which are (1) [" ", " ", " ", " "], (2) [" ", " ", " ", " "], (3) [" ", " ", " ", " "], (4) [" ", " ", " ", " "], (5) [" ", " ", " ", " "], (6) [" ", " ", " ", " "], (7) [" ", " ", " ", " "], and (8) [" ", " "]. The four vowel letters are " ", " ", " ", and " ".…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Gao

et al. 2023

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

Sentence Boundary Disambiguation (SBD) is a preprocessing step for natural language processing.Segmenting text into sentences is essential for Deep Learning (DL) and pretraining language models. Tibetan punctuation marks may involve ambiguity about the sentences' beginnings and endings. Hence, the ambiguous punctuation marks must be distinguished, and the sentence structure must be correctly encoded in language models. This study proposed a component-level Tibetan SBD approach based on the DL model. The models can reduce the error amplification caused by word segmentation and part-of-speech tagging. Although most SBD methods have only considered text on the left side of punctuation marks, this study considers the text on both sides. In this study, 465 669 Tibetan sentences are adopted, and a Bidirectional Long Short-Term Memory (Bi-LSTM) model is used to perform SBD. The experimental results show that the F1-score of the Bi-LSTM model reached 96%, the most efficient among the six models. Experiments are performed on low-resource languages such as Turkish and Romanian, and high-resource languages such as English and German, to verify the models' generalization.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Gao

et al. 2023

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…This thesis studies two approaches to improve the lexical input feature space for SUD systems. The first approach proposes a PMI-based feature selection method to select only informative features for a shallow learning (CRF-based) SUD system [22]. The second approach examines various strategies to train word embedding models which are used to extract feature for a deep learning based SUD system [23].…”

Section: Contributionsmentioning

confidence: 99%

“…The novel PMI-based feature selection method [22]: The approach investigates the effect of contextual features to the SUD task and proposes a feature selection method using PMI to reduce the input feature space. Here, the surrounding words of each label being predicted are used as the contextual features to the SUD system.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Sentence unit detection using lexical information for automatic speech recognition transcripts

Ho¹

View full text Add to dashboard Cite

This thesis studies Sentence Unit Detection (SUD) that uses lexical information for Automatic Speech Recognition (ASR) transcripts. An SUD system detects the presence or absence of a sentence unit boundary at every word boundaries of an unpunctuated text document. A transcript produced by an ASR system from a given speech audio input is an unpunctuated text document. Hence, SUD for ASR transcripts will increase the readability and enable the usability of ASR transcripts in downstream text processing applications, such as machine translation and option mining, which require sentence unit boundary information in their input. As usually being a post-processing application for ASR transcripts, an SUD system may benefit from two major information sources, i.e., the prosodic information obtained from the audio input and the lexical information obtained from the ASR transcripts, for its model learning. In this thesis, we chose to use only lexical information because it has shown to be essential for SUD research as compared to the other. Moreover, using lexical information takes an advantage of the massive available lexical data sources from the Internet. Additionally, SUD models that use only lexical information are usually much less complex as compared to the models that employ multiple information sources. To improve an SUD system performance, one possible direction is to improve the quality of input features. Previous studies usually paid more attention on finding new input features to enrich and enlarge the existing feature space rather than refining it for a better quality. This tendency might lead to feature redundancy and unnecessarily increase the model complexity. This thesis addresses the lack in research by studying approaches that fine tune existing lexical features to SUD systems. Specifically, there are two proposed approaches: The first approach focuses on optimizing the existing features to reduces the input feature space size. The second study focuses on evaluating i Acknowledgement I want to express my sincerest thanks to my supervisor, Dr.Chng Eng Siong, for his invaluable support and guidance. I also thank all my colleagues in Temasek Laboratories @NTU and the SCSE Speech Recognition team, who are not only co-workers but also friends. Especially, I would like to thanks Dr. Chong Tze Yuang, Ms. Vu Thi Ly, Mr. Xu Chenglin, Ms. Aditi Ghosh and Mr. Can Duy Cat for the help and support they have given to me. Last but not least, thanks to my family for their unconditional love, support, and encouragement. Without them, my research life would be much harder. I also want to give a special thanks to my husband and my little girl for always being beside me, whatever and whenever, the situation.

show abstract

LSTM-based Turn-taking Estimation Model using Lexical/Prosodic Contents and Dialog History

Liu¹,

Ishi²,

Ishiguro

2019

Transactions of the Japanese Society for Artificial Intelligenc

View full text Add to dashboard Cite

keywords: turn-taking, word embedding, prosody, dialog history, LSTM Summary A natural conversation involves rapid exchanges of turns while talking. Taking turns at appropriate timing or intervals is a requisite feature for a dialog system as a conversation partner. We propose a Recurrent Neural Network (RNN) based model that takes the current utterance and the dialog history as its input to classify utterances into turn-taking related classes and estimates the turn-taking timing. The dialog history is represented by a sequence of speaker-specified joint embedding of lexical and prosodic contents. To this end, we trained a neural network to embed the lexical and the prosodic contents into a joint embedding space. To learn meaningful embedding spaces, the prosodic feature sequence from each single utterance is mapped into a fixed-dimensional space using RNN and combined with utterance lexical embedding. These joint embeddings are then shifted to different parts of embedding spaces according to the speakers. Finally, the speaker-specified joint embeddings are used as the input of our proposed model. We tested this model on a spontaneous conversation dataset and confirmed that it outperformed conventional models that use lexical/prosodic features and dialog history without speaker information.

show abstract

Improving Efficiency of Sentence Boundary Detection by Feature Selection

Cited by 3 publications

References 9 publications

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

A Tibetan Sentence Boundary Disambiguation Model Considering the Components on Information on Both Sides of Shad

Sentence unit detection using lexical information for automatic speech recognition transcripts

LSTM-based Turn-taking Estimation Model using Lexical/Prosodic Contents and Dialog History

Contact Info

Product

Resources

About