On Chinese text retrieval

Nie, Jian‐Yun; Brisebois, Martin; Ren, Xiaobo

doi:10.1145/243199.243270

Cited by 41 publications

(32 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sophisticated IR systems do however include features which rely on semantics captured better in words than in bigrams: query expansion through synonyms, negated antonyms and other transformations based on thesaurus lookups; named entity extraction; question answering; spelling correction; semantic inference based on natural language processing (NLP) techniques. It might be possible to base relevance feedback on bigrams but this would have to be verified; it seems more natural to use words [16,22].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Text segmentation and Chinese site search

Zhou

Hawking

Thomas

2015

Proceedings of the 20th Australasian Document Computing Symposium

View full text Add to dashboard Cite

Automatic segmentation and overlapping bigrams are the most common methods for overcoming the lack of explicit word boundaries in Chinese text. Past studies have compared their effectiveness, but findings have been equivocal and site search has been little studied. We compare representatives of the two approaches using a 465,000 page crawl and test queries applicable to the university context. 503 pairs of result sets were judged by 56 Chinese students.Although there are differences on certain queries, we find no overall advantage to either method. To understand the merits of each approach, we analyze cases where they performed differently. Our analysis enumerates situations which favour segmentation, and those which favour bigrams. We observe that further improvements in segmentation accuracy will not improve retrieval effectiveness.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Tong et al [24] TREC Recall, AP, R-prec., P-R curves Equal Nie et al [16] 1270KB P-R curves Direct relationship Kwok [10] TREC AP, R-prec., P@N No direct relationship Kwok [9] TREC Rel. retrieved@1000, AP, R-prec., P@N Equal, no direct relationship Palmer and Burger [20] TREC AP, R-prec.…”

Section: Workmentioning

confidence: 99%

Text segmentation and Chinese site search

Zhou

Hawking

Thomas

2015

Proceedings of the 20th Australasian Document Computing Symposium

View full text Add to dashboard Cite

show abstract

“…It is found that approaches using either characters (bigrams) or words can lead to comparable retrieval effectiveness [6,11,12]. In [14], it is further found that the retrieval effectiveness using a character-based language model is highly competitive to, and on several collections, is even higher than that using words and bigrams.…”

Section: Related Workmentioning

confidence: 94%

Modeling Variable Dependencies between Characters in Chinese Information Retrieval

Shi

Nie

2010

Information Retrieval Technology

Self Cite

View full text Add to dashboard Cite

Abstract. Chinese IR can work on words and/or character n-grams. In previous studies, when several types of index are used, independence is usually assumed between them, which obviously is not true in reality. In this paper, we propose a model for Chinese IR that integrates different types of dependency between Chinese characters. The role of a pair of dependent characters in the matching process is variable, depending on the pair's ability to describe the underlying meaning and to retrieve relevant documents. The weight of the pair is learnt using SVM. Our experiments on TREC and NTCIR Chinese collections show that our model can significantly outperform most existing approaches. The results confirm the necessity to integrate dependent pairs of characters in Chinese IR and to use them according to their possible contribution to IR.

show abstract

“…In (Wechsler, 2000), there is also investigation on the use of subword phonemes for retrieving spoken documents in German and English. For information retrieval in Chinese, there has been some work on word segmentation for Chinese IR (Nie and Brisebois, 1996) as well as investigation on the use of words or character ngrams as indexing units for Chinese textual IR (Kwok, 1997 andRen, 1999). Subword-based indexing is useful for information retrieval in Chinese because all textual documents are made up of a finite number of characters.…”

Section: Subword-based Retrievalmentioning

confidence: 99%

Multi-Scale Spoken Document Retrieval for Cantonese Broadcast News

Meng

Ching

2004

International Journal of Speech Technology

View full text Add to dashboard Cite

This paper presents the application of a multi-scale paradigm to Chinese spoken document retrieval (SDR) for improving retrieval performance. Multi-scale refers to the use of both words and subwords for retrieval. Words are basic units in a language that carry lexical meaning and subword units (such as phonemes, syllables or characters) are building components for words. Retrieval using subword indexing units is found to perform better than words because of the robustness of subword units to out-of-vocabulary (OOV) words during speech recognition and ambiguities in word segmentation. Experimental results have demonstrated that subword bigrams can bring improvement in retrieval performance over words (~9.56%). Application of multi-scale fusion to SDR aims at combining the lexical information of words and the robustness of subwords. This work presents the first detailed investigation for a Cantonese broadcast news retrieval task using two different multi-scale fusion approaches: pre-retrieval fusion and post-retrieval fusion.Multi-scale retrieval using both words and syllable bigrams achieve improvement in retrieval performance (~1.90%) over retrieval on the composite scales.

show abstract

On Chinese text retrieval

Cited by 41 publications

References 9 publications

Text segmentation and Chinese site search

Text segmentation and Chinese site search

Modeling Variable Dependencies between Characters in Chinese Information Retrieval

Multi-Scale Spoken Document Retrieval for Cantonese Broadcast News

Contact Info

Product

Resources

About