A Length-variable Feature Code Based Fuzzy Duplicates Elimination Approach for Large Scale Chinese WebPages

Abstract

<p class="Abstract">Most of the existing Chinese webpage duplicate elimination approaches do not focus on noisy and fuzzy duplicates elimination. In this paper, we propose an efficient and noise-tolerant Chinese webpage duplicate elimination approach based on Length-variable Feature Code. First, an Independent Extraction Unit is defined to eliminate the impact of short paragraphs on feature code extraction. Then the concept of repeatability is introduced by using the longest common substring to enhance the noise tolerant capability. Experimental results on 10 million webpage dataset show that the proposed approach can efficiently deal with duplicates from massive WebPages with the duplicate elimination precision of 99.03%.</p

User Clustering Topic Recommendation Algorithm Based on Two Phase in the Social Network

“…Social network shows the trend of rapid development with the popularity of Internet users, not only the number of users with explosive growth, but also its service form is also changing rapidly. In recent years, a large number of new social networking services constantly emerging, of which weak relationship between social network service of represented by sina weibo at home both and Facebook at abroad is becoming a major form of social network [1][2][3].Different from the traditional social network, due to the unidirectional of the weak relationship, based on the nodes of social network of weak relationship (that is, a one-way relationship) presents obvious heterogeneity characteristics, including a large number of users in a natural man as major body node (e.g., "zhangsan") and in the media, institutions and various sources as the main theme node (e.g., "the weather of Beijing", "south weekend", "popular video", etc.). Among them, the user nodes, usually as a message subscriber, one-way attention to a large number of topics node, the one-way subscription relationship, often based on the user tendency of interest for different types of theme; at the same time the user node often form a two-way relationship with other users, this is usually based on the user's real social relations [4][5][6].…”

A High-precision Duplicate Image Deduplication Approach

“…Deduplication can identify redundant data, eliminate all but one copy, and create local pointers to the information that users can access. This technology has been widespread concerned by industry and academia [2,3,4,5].…”