The study on Detecting Near-Duplicate WebPages

Abstract

Reprinting information among websites produces a great deal redundant WebPages. To improve search efficiency and user satisfaction, an algorithm to Detect near-Duplicate WebPages (DDW) is proposed. In the course of developing a near-duplicate detection system for a multi-billion page repository, we make two research contributions. First, we consider both syntactic and semantic information to present and compute documents' similarities. Second, after classifying web-pages into different categories, we index feature in each category then search for nearduplicates only in the same category. From Google searching results for 72 queries, we select 5835 nearduplicate WebPages manually. Then insert them into an existing collection which contains about 768,763 WebPages, as the test data. The experimental results demonstrate that our approach outperforms I-Match algorithms. In large-scale test, approximate linear time and space complexity are gotten.

A Length-variable Feature Code Based Fuzzy Duplicates Elimination Approach for Large Scale Chinese WebPages

“…However in practice there probably still are some noises left. For example, " [1] [2] [3] [ 我来说两句]", "相关专 题：法治在线节目实录" etc. These often occurred in the beginning of the texts; second, some of the ads and comments, which were added to the duplicated web pages, might not be removed completely during the purification.…”

“…The long text is: 在神七科研攻关中，哈尔滨工业大学承担着宇航员舱外宇航服的地 面实验系统，学院气动技术中心课题组负责"水平舱环控系统改造"和 "紧急复压系统"的研制。……而在神七发射前 38 分钟，上海的一名 科研人员拔掉地面连接神七的最后一个重要插座，成为最后一位撤离发 射架的人。 3 Experimental results of the extracted feature codes are listed in TABLE I. S A'A is the repeatability of the benchmark over the testing documents and S AA' is the repeatability of the testing documents over the benchmark.…”

“…Different kinds of approaches have been proposed for duplicates detection and elimination in recent years [2][3][4][5][6]. However most of them do not focus on noisy and fuzzy duplicates elimination.…”