Text mining: a new frontier for lossless compression

Witten, Ian H.; Bray, Z.; Mahoui, Malika; Teahan, B.

doi:10.1109/dcc.1999.755669

Cited by 40 publications

(37 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Compression methods have already found respectable application in various areas of text mining, as recently predicted by Witten et al [33]. Some of the areas include extraction of generic entities, token segmentation, acronym extraction [32] and text categorization [32,10].…”

Section: Compression Methods In Text Miningmentioning

confidence: 99%

“…Other applications of compression algorithms include identification of structured sources, where a structured source outputs specially formatted sequences, or even acronym finders where acronyms were found as the occurrences of typically capital letters with significantly different letter or language statistics [32,33].…”

Section: Compression Methods In Text Miningmentioning

confidence: 99%

See 1 more Smart Citation

Using Compression to Identify Classes of Inauthentic Texts

Dalkılıç¹,

Clark²,

Costello³

et al. 2006

Proceedings of the 2006 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

Recent events have made it clear that some kinds of technical texts, generated by machine and essentially meaningless, can be confused with authentic, technical texts written by humans. We identify this as a potential problem, since no existing systems for, say the web, can or do discriminate on this basis. We believe that there are subtle, short-and longrange word or even string co-occurrences extant in human texts, but not in many classes of computer generated texts, that can be used to discriminate based on meaning. In this paper we employ the universal lossless source coding algorithms to generate features in a high-dimensional space and then apply support vector machines to discriminate between the classes of authentic and inauthentic texts. Compression profiles for the two kinds of text are distinctthe authentic texts being bounded by various classes of more compressible or less compressible texts that are computer generated. This in turn led to the high prediction accuracy of our models which support our conjecture that there exists a relationship between meaning and compressibility. Our results show that the learning algorithm based upon the compression profile outperformed standard term-frequency text categorization schemes on several non-trivial classes of inauthentic texts.

show abstract

Section: Compression Methods In Text Miningmentioning

confidence: 99%

Section: Compression Methods In Text Miningmentioning

confidence: 99%

Using Compression to Identify Classes of Inauthentic Texts

Dalkılıç¹,

Clark²,

Costello³

et al. 2006

Proceedings of the 2006 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

show abstract

“…To do so requires new techniques of information mining. We are working on these [13], but they are still in their infancy and we have not begun to apply them to harvesting information about music. Moreover, items should be cross-referenced to other musical sites.…”

Section: Browsing and Searchingmentioning

confidence: 99%

Towards a digital library of popular music

Bainbridge

Nevill-Manning

Witten

et al. 1999

Proceedings of the Fourth ACM Conference on Digital Libraries

View full text Add to dashboard Cite

Digital libraries of music have the potential to capture popular imagination in ways that more scholarly libraries cannot. We are working towards a comprehensive digital library of musical material, including popular music. We have developed new ways of collecting musical material, accessing it through searching and browsing, and presenting the results to the user. We work with different representations of music: facsimile images of scores, the internal representation of a music editing program, page images typeset by a music editor, MIDI files, audio files representing sung user input, and textual metadata such as title, composer and arranger, and lyrics. This paper describes a comprehensive suite of tools that we have built for this project. These tools gather musical material, convert between many of these representations, allow searching based on combined musical and textual criteria, and help present the results of searching and browsing. Although we do not yet have a single fully-blown digital music library, we have built several exploratory prototype collections of music, some of them very large (100,000 tunes), and critical components of the system have been evaluated.

show abstract

“…PST have been applied to gene/protein sequence clustering [66] and spam filtering [45]. Researchers have tried to do text classification by using PPM, PPM* and other text compression methods [5,18,43,58,64].…”

Section: The Generative Approachmentioning

confidence: 99%

Extracting key-substring-group features for text classification

Zhang

Lee

2006

Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

In many text classification applications, it is appealing to take every document as a string of characters rather than a bag of words. Previous research studies in this area mostly focused on different variants of generative Markov chain models. Although discriminative machine learning methods like Support Vector Machine (SVM) have been quite successful in text classification with word features, it is neither effective nor efficient to apply them straightforwardly taking all substrings in the corpus as features. In this paper, we propose to partition all substrings into statistical equivalence groups, and then pick those groups which are important (in the statistical sense) as features (named keysubstring-group features) for text classification. In particular, we propose a suffix tree based algorithm that can extract such features in linear time (with respect to the total number of characters in the corpus). Our experiments on English, Chinese and Greek datasets show that SVM with key-substring-group features can achieve outstanding performance for various text classification tasks.

show abstract

Text mining: a new frontier for lossless compression

Cited by 40 publications

References 5 publications

Using Compression to Identify Classes of Inauthentic Texts

Using Compression to Identify Classes of Inauthentic Texts

Towards a digital library of popular music

Extracting key-substring-group features for text classification

Contact Info

Product

Resources

About