Comparison of Overlap Detection Techniques

Monostori, Krisztián; Finkel, Raphael A.; Zaslavsky, Arkady; Hodász, Gábor; Pataki, Máté

doi:10.1007/3-540-46043-8_4

Cited by 25 publications

(22 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different methods and approaches have been used to tackle this issue of similarities between documents using semantically, syntactical or semantic features. Semantic similarity received less attention for the inherent difficulties of representing semantics and the limitations on assessment coverage of user studies [54] [72]. Commonly used methods for determination of similarity include fingerprinting [21], Information Retrieval [28] and other hybrid techniques [24] [44].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Extractive Summarization Using Structural Syntax, Term Expansion and Refinement

Elhadi¹

2017

IJIS

View full text Add to dashboard Cite

This paper investigates a procedure developed and reports on experiments performed to studying the utility of applying a combined structural property of a text's sentences and term expansion using WordNet [1] and a local thesaurus [2] in the selection of the most appropriate extractive text summarization for a particular document. Sentences were tagged and normalized then subjected to the Longest Common Subsequence (LCS) algorithm [3] [4] for the selection of the most similar subset of sentences. Calculated similarity was based on LCS of pairs of sentences that make up the document. A normalized score was calculated and used to rank sentences. A selected top subset of the most similar sentences was then tokenized to produce a set of important keywords or terms. The produced terms were further expanded into two subsets using 1) WorldNet; and 2) a local electronic dictionary/thesaurus. The three sets obtained (the original and the expanded two) were then re-cycled to further refine and expand the list of selected sentences from the original document. The process was repeated a number of times in order to find the best representative set of sentences. A final set of the top (best) sentences was selected as candidate sentences for summarization. In order to verify the utility of the procedure, a number of experiments were conducted using an email corpus. The results were compared to those produced by human annotators as well as to results produced using some basic sentences similarity calculation method. Produced results were very encouraging and compared well to those of human annotators and Jacquard sentences similarity.

show abstract

Section: Related Workmentioning

confidence: 99%

“…The combined use of syntactical POS tagging and text processing methods for the purpose of text similarity calculations and its applications was used in this recent work [72]- [77]. It was based on the intuition that similar (exact) documents would have similar (exact) syntactical structures.…”

Section: Related Workmentioning

confidence: 99%

Extractive Summarization Using Structural Syntax, Term Expansion and Refinement

Elhadi¹

2017

IJIS

View full text Add to dashboard Cite

show abstract

“…Other mostly non-semantically oriented techniques have received more attention. These include fingerprinting [29], IR [6] and many hybrid techniques [5,13,14]. In information retrieval models, more emphasis is put on representing documents by their words and word frequencies.…”

Section: Related Workmentioning

confidence: 99%

“…It spans many fields of research including, among many other applications, copy/ near-copy detection [1,2], plagiarism [3][4][5], Information Retrieval (IR) [6,7] and computational biology [8][9][10]12]. Many such applications employee a combination of techniques and apply to multidisciplinary fields [13,14,15]. This paper reports on a work that investigated how related Arabic documents can be treated as modified versions of one another using edit operations [16][17][18].…”

Section: Introductionmentioning

confidence: 99%

Arabic Text Copy Detection using Full, Reduced and Unique Syntactical Structures

Elhadi¹

2016

IJCA

View full text Add to dashboard Cite

This paper reports on work performed to investigate the use of a combined Part of Speech (POS) tagging and a minimum edit operations algorithm to determine the level of similarity between pairs of Arabic text documents. The level of similarity can be used as an indication of duplication in full or in part of the document's content. Text is first converted into POS tags that are then fed to the string similarity algorithm to determine the similarity of pairs of documents. A normalized score is calculated and used to rank documents. Documents ranked higher than some selected threshold are considered similar and can be near or complete duplicate. The performed experiments compare results based on the use of a set of selected common subsequences that are the results of translation of text into a sequence of syntactical units. The strings are first produced using full-text (FULL). These are further refined to produce a REDUCED; where repeated consecutive characters are reduced to a single character and a number, and more refined to produce a UNIQUE string; where all repeating characters are replaced by a single character. Syntactical features of the text were used as a structural representation of the documents' content. Results obtained from the experiments using the FULL, the REDUCED and the UNIQUE POS-strings showed a clear advantage over the use of the plain text in terms of reduced string size while maintaining the same discrimination power. In particular the unique (most-reduced) string has shown quite comparable results to the reduced, the full and the actual text string.

show abstract

“…These include plagiarism (Braumoeller, 2001;Monostori, 2002;Cook, 2002;Hoad, 2003;Gilbert, 2003;Pecorari, 2003;Chen, 2004;Bao, 2004), duplicate/ redundant publication (Doherty, 1996;Jefferson, 1998;Schein, 2001;Bailey, 2002;Von Elm, 2004;Mojon-Azzi, 2004;Gwilym, 2004), text/ document clustering (Maderlechner, 1997;Atlam, 2003;Dobrynin, 2004;Shin, 2004;Bansal, 2004), and information retrieval (Salton, 1991;Hui, 2004;Leuski, 2004;Muresan, 2004;Chang, 2004). These studies have shown that, in general, identifying similar documents through concept matching is quite difficult.…”

Section: Introductionmentioning

confidence: 99%