Text Segmentation Techniques: A Critical Review

Pak, Irina; Teh, Phoey Lee

doi:10.1007/978-3-319-66984-7_10

Cited by 35 publications

(19 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Segmentation includes methods that break a document down into independent and minimal textual components which are usually called segments or tokens. 62 A text segment is defined as a contiguous piece of text that is linked to itself but largely disconnected from the adjacent text. 63…”

Section: Definitions and Terminologymentioning

confidence: 99%

“…A token is an instance of a sequence of characters that are semantically grouped together. 64 Some literature, such as Pak and Teh, 62 considers tokenization as a sub function of segmentation. To some extent, we agree they overlap and could be used interchangeably, however in the context of this paper, we refer to tokenization and segmentation as two stages, as defined here.…”

Section: Definitions and Terminologymentioning

confidence: 99%

See 1 more Smart Citation

VNLP: Visible natural language processing

Alharbi

Roach

Cheesman

et al. 2021

Information Visualization

View full text Add to dashboard Cite

In general, Natural Language Processing (NLP) algorithms exhibit black-box behavior. Users input text and output are provided with no explanation of how the results are obtained. In order to increase understanding and trust, users value transparent processing which may explain derived results and enable understanding of the underlying routines. Many approaches take an opaque approach by default when designing NLP tools and do not incorporate a means to steer and manipulate the intermediate NLP steps. We present an interactive, customizable, visual framework that enables users to observe and participate in the NLP pipeline processes, explicitly manipulate the parameters of each step, and explore the result visually based on user preferences. The visible NLP (VNLP) pipeline design is then applied to a text similarity application to demonstrate the utility and advantages of a visible and transparent NLP pipeline in supporting users to understand and justify both the process and results. We also report feedback on our framework from a modern languages expert.

show abstract

Section: Definitions and Terminologymentioning

confidence: 99%

Section: Definitions and Terminologymentioning

confidence: 99%

VNLP: Visible natural language processing

Alharbi

Roach

Cheesman

et al. 2021

Information Visualization

View full text Add to dashboard Cite

show abstract

“…e structured data can be transformed into mathematical problems, and word segmentation is the first step in this transformation [14]. e basic unit of English text is a word [15]; therefore, standard word segmentation techniques are divided into three steps:…”

Section: Related Workmentioning

confidence: 99%

An Efficient Minimal Text Segmentation Method for URL Domain Names

Zhu

et al. 2021

Scientific Programming

View full text Add to dashboard Cite

Text segmentation of the URL domain name is a straightforward and convenient method to analyze users’ online behaviors and is crucial to determine their areas of interest. However, the performance of popular word segmentation tools is relatively low due to the unique structure of the website domain name (such as extremely short lengths, irregular names, and no contextual relationship). To address this issue, this paper proposes an efficient minimal text segmentation (EMTS) method for URL domain names to achieve efficient adaptive text mining. We first designed a targeted hierarchical task model to reduce noise interference in minimal texts. We then presented a novel method of integrating conflict game into the two-directional maximum matching algorithm, which can make the words with higher weight and greater probability to be selected, thereby enhancing the accuracy of recognition. Next, Chinese Pinyin and English mapping were embedded in the word segmentation rules. Besides, we incorporated a correction factor that considers the text length into the F1-score to optimize the performance evaluation of text segmentation. The experimental results show that the EMTS yielded around 20 percentage points improvement with other word segmentation tools in terms of accuracy and topic extraction, providing high-quality data for the subsequent text analysis.

show abstract

“…Tasks such as word tagging and tokenizing are done in many different languages, including Arabic [9], Hebrew [10], German [11], Urdu [12], Burmese [13], Russian [14], Chinese [15] and Swedish [16]. In other words, the process of text segmentation involved in these studies has been used in many different languages for text analysis [17].…”

Section: Introductionmentioning

confidence: 99%

Compilation of malay criminological terms from online news

Lee

Teh

Lau

et al. 2019

IJEECS

Self Cite

View full text Add to dashboard Cite

<p>A Malay language corpus has been established by the Institute of Language and Literature (Dewan Bahasa dan Pustaka, DBP in Malaysia). Most of the past research on the Malay language corpus has focused on the description, lexicography and translation of the Malay language. However, in the existing literature, there is no list of Malay words that categorizes crime terminologies. This study aims to fill that linguistic gap. First, we aggregated the most frequently used crime terminology words from Malaysian online news sources. Five hundred crime-related words were compiled. No automatic machines were in the initial process, but they were subsequently used to verify the data. Four human coders were used to validate the data and ensure the originality of the semantic understanding of the Malay text. Finally, major crime terminologies were outlined from a set of keywords to serve as taggers in our solution. The ultimate goal of this study is to provide a corpus for forensic linguistics, police investigations, and general crime research. This study has established the first corpus of a criminological text in the Malay language.</p>

show abstract

Text Segmentation Techniques: A Critical Review

Cited by 35 publications

References 45 publications

VNLP: Visible natural language processing

VNLP: Visible natural language processing

An Efficient Minimal Text Segmentation Method for URL Domain Names

Compilation of malay criminological terms from online news

Contact Info

Product

Resources

About