Statistical keyword detection in literary corpora

Herrera, Juan P; Pury, Pedro A.

doi:10.1140/epjb/e2008-00206-x

Cited by 65 publications

(79 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Because the distribution of 'Carmylle' is much more irregular than the distribution of 'feel', the former takes a much higher value of intermittency, as defined in equation (5). The burstiness revealed by 'Carmylle' also suggests that this word represents a relevant concept in the book [31]. An important property of the intermittency measurement is that it does not correlate with the frequency (see figure 3).…”

Section: Intermittencymentioning

confidence: 99%

Authorship recognition via fluctuation analysis of network topology and word intermittency

Amancio¹

2015

J. Stat. Mech.

View full text Add to dashboard Cite

Abstract. Statistical methods have been widely employed in many practicalnatural language processing applications. More specifically, complex network concepts and methods from dynamical systems theory have been successfully applied to recognize stylistic patterns in written texts. Despite the large number of studies devoted to representing texts with physical models, only a few studies have assessed the relevance of attributes derived from the analysis of stylistic fluctuations. Because fluctuations represent a pivotal factor for characterizing a myriad of real systems, this study focused on the analysis of the properties of stylistic fluctuations in texts via topological analysis of complex networks and intermittency measurements. The results showed that different authors display distinct fluctuation patterns. In particular, it was found that it is possible to identify the authorship of books using the intermittency of specific words. Taken together, the results described here suggest that the patterns found in stylistic fluctuations could be used to analyze other related complex systems. Furthermore, the discovery of novel patterns related to textual stylistic fluctuations indicates that these patterns could be useful to improve the state of the art of many stylistic-based natural language processing tasks.

show abstract

Section: Intermittencymentioning

confidence: 99%

Authorship recognition via fluctuation analysis of network topology and word intermittency

Amancio¹

2015

J. Stat. Mech.

View full text Add to dashboard Cite

show abstract

“…However, there are numerous situations where the comparison with a general database is not available or is not interesting: for instance, when authorship has to be attributed without previous knowledge of texts written by the potential authors. Here, we approach these problems by taking advantage of the finding that words are unevenly distributed not only across documents but also within them [16][17][18][21][22][23][24]. The quantification of the uneven distribution of words has been proposed based on measures commonly used by physicists [16,24].…”

Section: Intermittency Measurementsmentioning

confidence: 99%

“…The usefulness of the latter approach stems from the finding that topical words are unevenly distributed along the text when compared with a random process or function words. This observation can be quantitatively investigated using different analogies and measures familiar to the communities of statistical physics and dynamical systems, including level statistics [22,24], burstiness [17][18][19], entropy [21] and intermittency measures [20]. The author dependence of the features mentioned above has been noted [14,17], but very little work has so far been devoted to quantifying the extent of this dependence and to testing its usefulness for the automatic detection of authors.…”

Section: Introductionmentioning

confidence: 99%

Comparing intermittency and network measurements of words and their dependence on authorship

et al. 2011

View full text Add to dashboard Cite

Abstract. Many features of texts and languages can now be inferred from statistical analyses using concepts from complex networks and dynamical systems. In this paper, we quantify how topological properties of word cooccurrence networks and intermittency (or burstiness) in word distribution depend on the style of authors. Our database contains 40 books by eight authors who lived in the nineteenth and twentieth centuries, for which the following network measurements were obtained: the clustering coefficient, average shortest path lengths and betweenness. We found that the two factors with stronger dependence on authors were skewness in the distribution of word intermittency and the average shortest paths. Other factors such as betweenness and Zipf's law exponent show only weak dependence on authorship. Also assessed was the contribution from each measurement to authorship recognition using three machine learning methods. The best performance was about 65% accuracy upon combining complex networks and intermittency features with the nearestneighbor algorithm of automatic authorship. From a detailed analysis of the interdependence of the various metrics, it is concluded that the methods used here are complementary for providing short-and long-scale perspectives on texts, which are useful for applications such as the identification of topical words and information retrieval.

show abstract

“…In 2008, J.P. Herrera et al tackled the problem of finding and ranking the relevant words of a document by using statistical information referring to the spatial use of the words [6]. Shannon's entropy of information was used for automatic keyword extraction.…”

Section: Some Linguistic Properties Of Keyphrasesmentioning

confidence: 99%

Keyword and Keyphrase Extraction Techniques: A Literature Review

Siddiqi¹,

Sharan²

2015

IJCA

177

View full text Add to dashboard Cite

In this paper we present a survey of various techniques available in text mining for keyword and keyphrase extraction. Keywords and keyphrases are very useful in analyzing large amount of textual material quickly and efficiently search over the internet besides being useful for many other purposes. Keywords and keyphrases are set of representative words of a document that give high-level specification of the content for interested readers. They are used highly in the field of Computer Science especially in Information Retrieval and Natural Language Processing and can be used for index generation, query refinement, text summarization, author assistance, etc. We have also discussed some important feature selection metrics generally employed by researchers to rank candidate keywords and keyphrases according to their importance.

show abstract

Statistical keyword detection in literary corpora

Cited by 65 publications

References 16 publications

Authorship recognition via fluctuation analysis of network topology and word intermittency

Authorship recognition via fluctuation analysis of network topology and word intermittency

Comparing intermittency and network measurements of words and their dependence on authorship

Keyword and Keyphrase Extraction Techniques: A Literature Review

Contact Info

Product

Resources

About