Automatic Web Content Extraction by Combination of Learning and Grouping

Wu, Shanchan; Liu, Jerry; Fan, Jian

doi:10.1145/2736277.2741659

Cited by 29 publications

(13 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Next, create the feature Dom tree is created, and finally ,use the feature Dom tree for noise detection. Enhanced Dom tree and context features are also tested in [37] [38] [39] for noise detection.…”

Section: B Page Noise Reductionmentioning

confidence: 99%

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

Gong

Zhang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Usually, in addition to the main content, web pages contain additional information in the form of noise, such as navigation elements, sidebars and advertisements. This kind of noise has nothing to do with the main content, it will affect the tasks of data mining and information retrieval so that the sensor will be damaged by the wrong data and interference noise. Because of the diversity of web page structure, it is a challenge to detect relevant information and noise in order to improve the true reliability of sensor networks. In this paper, we propose a visual block construction method based on page type conversion (VB-PTC). This method uses a combination of site-level noise reduction based on hashtree and page-level noise reduction based on linked clusters to eliminate noise in web articles, and it successfully converts multi-record complex pages to multi-record simple pages, effectively simplifying the rules of visual block construction. In the aspect of multi-record content extraction, according to the characteristics of different fields, we use different extraction methods, combined with regular expression, natural language processing and symbol density detection methods which greatly improves the accuracy of multi-record content extraction. VB-PTC can be effectively used for information retrieval, content extraction and page rendering tasks.

show abstract

Section: B Page Noise Reductionmentioning

confidence: 99%

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

Gong

Zhang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Wu et al [47] proposed a machine learning model using DOM tree node features such as position, area, font, text and tag properties to select and group content related nodes and their children. In their recent paper, Vogels et al [42] presented an algorithm combining a hidden markov model in and a convolutional neural networks (CNNs).…”

Section: Related Work 61 Content Extractionmentioning

confidence: 99%

SpeedReader: Reader Mode Made Fast and Private

Ghasemisharif

Snyder²,

Auçinas

et al. 2019

The World Wide Web Conference

View full text Add to dashboard Cite

Most popular web browsers include "reader modes" that improve the user experience by removing un-useful page elements. Reader modes reformat the page to hide elements that are not related to the page's main content. Such page elements include site navigation, advertising related videos and images, and most JavaScript. The intended end result is that users can enjoy the content they are interested in, without distraction.In this work, we consider whether the "reader mode" can be widened to also provide performance and privacy improvements. Instead of its use as a post-render feature to clean up the clutter on a page we propose SpeedReader as an alternative multistep pipeline that is part of the rendering pipeline. Once the tool decides during the initial phase of a page load that a page is suitable for reader mode use, it directly applies document tree translation before the page is rendered.Based on our measurements, we believe that SpeedReader can be continuously enabled in order to drastically improve end-user experience, especially on slower mobile connections. Combined with our approach to predicting which pages should be rendered in reader mode with 91% accuracy, it achieves drastic speedups and bandwidth reductions of up to 27× and 84× respectively on average. We further find that our novel "reader mode" approach brings with it significant privacy improvements to users. Our approach effectively removes all commonly recognized trackers, issuing 115 fewer requests to third parties, and interacts with 64 fewer trackers on average, on transformed pages.

show abstract

“…Web content extraction is very well investigated in the literature [1,2,3,4,5,6,7,8,9,10,11,12]. Many of these approaches apply techniques based on certain heuristics, machine learning or site specific solutions like rule based content extraction, DOM tree parsing, Text graph or Link Graph or vision based models, or NLP features like N −grams or shallow text features like number of tokens, average sentence length and so on.…”

Section: Related Workmentioning

confidence: 99%

“…In this work, a set of relevant features is selected for each text block in the HTML document and then using a Support Vector Machine (SVM) classifier, each text block is classified as either content block or non-content block. Most recently, Wu et al [12] formulated the content identification problem as a DOM tree node selection problem. Using multiple features from DOM node properties, a machine learning model is trained and a set of candidate nodes is selected based on the learning model.…”

Section: Related Workmentioning

confidence: 99%

An Approach to Content Extraction from Scientific Articles using Case-Based Reasoning

Prasath¹,

Öztürk²

2016

RCS

View full text Add to dashboard Cite

In this paper, we present an efficient approach for content extraction of scientific papers from web pages. The approach uses an artificial intelligence method, Case-Based Reasoning(CBR), that relies on the idea that similar problems have similar solutions and hence reuses past experiences to solve new problems or tasks. The key task of content extraction is the classification of HTML tag sequences where the sequences representing navigation links, advertisements and, other non-informative content are not of interest when the goal is to extract scientific contributions. Our method learns from each experience with the tag sequence classification episode and stores these in the case base. When a new tag sequence needs to be classified, the system checks its case base to see whether a similar tag was experienced before in order to reuse it for content extraction. If the tag sequence is completely new, then it uses the proposed algorithm that relies on two assumptions related to the distribution of various tag sequences occurring in the page, and the similarity of the tag sequences with respect to their structure in terms of levels of the tags. Experimental results show that the proposed approach efficiently extracts content information from scientific articles.

show abstract

Automatic Web Content Extraction by Combination of Learning and Grouping

Cited by 29 publications

References 23 publications

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

SpeedReader: Reader Mode Made Fast and Private

An Approach to Content Extraction from Scientific Articles using Case-Based Reasoning

Contact Info

Product

Resources

About