Effectual Web Content Mining using Noise Removal from Web Pages

Sivakumar, P.

doi:10.1007/s11277-015-2596-7

Cited by 21 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Jaccard coefficient [29] was, for example, used by Sivakumar [30] in 2015. The purpose of the study was to ultimately improve search results by comparing blocks of text within web pages, as well as identifying and removing duplicate advertisements, headers, and other recurring features present in web sites.…”

Section: Related Workmentioning

confidence: 99%

Scholarly Context Adrift: Three out of Four URI References Lead to Changed Content

et al. 2016

View full text Add to dashboard Cite

Increasingly, scholarly articles contain URI references to “web at large” resources including project web sites, scholarly wikis, ontologies, online debates, presentations, blogs, and videos. Authors reference such resources to provide essential context for the research they report on. A reader who visits a web at large resource by following a URI reference in an article, some time after its publication, is led to believe that the resource’s content is representative of what the author originally referenced. However, due to the dynamic nature of the web, that may very well not be the case. We reuse a dataset from a previous study in which several authors of this paper were involved, and investigate to what extent the textual content of web at large resources referenced in a vast collection of Science, Technology, and Medicine (STM) articles published between 1997 and 2012 has remained stable since the publication of the referencing article. We do so in a two-step approach that relies on various well-established similarity measures to compare textual content. In a first step, we use 19 web archives to find snapshots of referenced web at large resources that have textual content that is representative of the state of the resource around the time of publication of the referencing paper. We find that representative snapshots exist for about 30% of all URI references. In a second step, we compare the textual content of representative snapshots with that of their live web counterparts. We find that for over 75% of references the content has drifted away from what it was when referenced. These results raise significant concerns regarding the long term integrity of the web-based scholarly record and call for the deployment of techniques to combat these problems.

show abstract

Section: Related Workmentioning

confidence: 99%

Scholarly Context Adrift: Three out of Four URI References Lead to Changed Content

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Several hashing techniques such as minhash [24], simhash [25] and hybrid hash [26] techniques are widely used to eliminate the noises present in the web pages and also extracting the duplicates and near-duplicate blocks present in the web pages. Noisy Data Cleaner (NDC) algorithm [27] was introduced to extract core content and to eliminate the noises present in the web pages.…”

Section: Literature Reviewmentioning

confidence: 99%

“…The comparative analysis has also been made with the proposed method by comparing it with other existing methods such as N-gram approach [7], sentence level features with fingerprints (SLF-FP) [22], SimSeerX [23], enhanced weighted approach [19], Simhash [25], hybrid hash [26], and NDC algorithm [27]. The comparative analysis for the proposed model and the other mentioned exiting model is carried out and 100 documents from the three datasets DS1, DS2, and DS3 are taken for the analysis having varied number of RD and DD documents.…”

Section: Performance Analysismentioning

confidence: 99%

A Novel Technique Using Multiple K-Shingling Based Weighted Dissimilarity Score for Web Content Outlier Mining

Khan¹,

Ahmed²,

Almistarihi³

2019

IJIES

View full text Add to dashboard Cite

The technological evolution of Internet and Web along with several applications leads to the problem of redundancy as the documents are unanimously forwarded and are stored in several servers and platforms. Recently, not only duplicate documents but also near-duplicate documents affect the performance of the search results. The main objective of this paper is to provide significant documents by eliminating the redundancy and near redundancy documents present in the web search results. The proposed model comprises of two phases such as pre-processing phase and dissimilarity computation phase. For dissimilarity computation, the proposed model employs multiple kshingling based dissimilarity score to identify the duplicate and near-duplicate documents which are considered as the outliers present in the set of input web documents. The proposed model has been evaluated using several experimental analysis. As there are no real datasets available for duplicate detection, datasets have been created and the performance evaluation is carried out with the created datasets. Several statistical analysis has been made wherein the average specificity, sensitivity, precision, and accuracy are 87%, 93%, 80%, and 92% respectively. The comparative analysis has also been made with various existing methods, in which the proposed model provides better results than existing methods in removing near-duplicates. The proposed multiple k-shingling based weighted dissimilarity model effectively detects the duplicates and near-duplicates when the number of outliers is minimum.

show abstract

“…are discussed. Duplicate contents and noise content as per block importance are removed [16]. Global and local noises are removed [17].…”

Section: Noise Removal From Web Pagesmentioning

confidence: 99%

Enhanced Methods of Mobile Cache Sharing and Pre-fetching for Required Web Contents

Bhomini*¹,

J.S²

2019

IJRTE

View full text Add to dashboard Cite

Availability of content over the web is increasing exponentially. The demand for content by users is also increasing rapidly. The problem of making the right content available to user at the right time will continue to be a crucial issue. As variety of contents are available and variety of users are involved, there is no single way of matching the availability versus need and deliver content instantly especially in a limited mobile environment. Hence a hybrid method is proposed in this paper by combining the different techniques such as caching, pre-fetching and cache sharing with noise reduction to improve the overall performance of mobile for optimal cache memory utilisation, efficient bandwidth utilization, network traffic reduction and latency reduction. Efficiency of mobile caching and pre-fetching is improved using Enhanced Bloom Filter technique and data is shared among cooperative users by establishing a voluntary hub. The unwanted contents in the web page can be considered as noise which is removed when storing the web pages in cache or pre-fetch area. The success of the proposed method greatly depends on the hit ratio of contents rendered locally rather than getting it from server. In order to reduce server hits, sharing the contents of cache and pre-fetch area amongst mobile users is effective. Whenever any user requires new content, even if it is not available in browser cache or local cache of that user, the content can be rendered from the cache or pre-fetch area of collaborative mobile users rather than hitting the server. This hybrid cooperative cache sharing and pre-fetching for accessing the required contents improve the overall performance and hit ratio than the existing methods.

show abstract

Effectual Web Content Mining using Noise Removal from Web Pages

Cited by 21 publications

References 12 publications

Scholarly Context Adrift: Three out of Four URI References Lead to Changed Content

Scholarly Context Adrift: Three out of Four URI References Lead to Changed Content

A Novel Technique Using Multiple K-Shingling Based Weighted Dissimilarity Score for Web Content Outlier Mining

Enhanced Methods of Mobile Cache Sharing and Pre-fetching for Required Web Contents

Contact Info

Product

Resources

About