Accurate and efficient general-purpose boilerplate detection for crawled web corpora

Schäfer, Roland

doi:10.1007/s10579-016-9359-2

Cited by 32 publications

(23 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A brief discussion of the benefits for BI of our approach as well as a sketch of the algorithm is presented in section 3.1. Differently from approaches like that in [30] our approach is not technology-specific and, thus, it is not supposed to become deprecated with the evolution of web technologies. Moreover, integrating our approach with our crawling ordering described in section 2.2, it is possible to reduce the storage requirements of crawling still enabling to restore the original webpages.…”

Section: Filteringmentioning

confidence: 97%

Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

Genovese¹,

Geraci²

2018

JSW

View full text Add to dashboard Cite

Business intelligence (BI) is the activity of extracting strategic information from big data. The benefits of this activity for enterprises span from the reduction of the operative costs due to a more sensible internal organization to a more productive and aware decision process. To be effective, BI relies heavily on the availability of a huge amount of (possibly high-quality) data. The steady decrease of costs for acquiring, storing and analyzing large knowledge bases has motivated big companies to invest in BI technologies. Until now, instead, SMEs (Small and Medium-sized Companies) are excluded from the benefits of BI because of their limited budget and resources. In this paper we show that a satisfactory BI activity is possible even in presence of a small budget. Our ultimate goal is not necessarily that of proposing novel solutions but providing the practitioners with a sort of hitchhiker's guide to a cost-effective web-based BI. In particular, we discuss how the Web can be used as a cheap yet reliable source of information where crawling, data cleaning and classification can be achieved using a limited amount of CPU, storage space and bandwidth.

show abstract

Section: Filteringmentioning

confidence: 97%

Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

Genovese¹,

Geraci²

2018

JSW

View full text Add to dashboard Cite

show abstract

“…The system uses a variety of text functions, such as the ratio of text to label, the ratio of anchor text to text, and the density of title, keywords. In addition, the noise element detection of MLP (Multi-Layer Perceptron) is also carried out [27]. The results show that the linguistic features of sentence length play a key role in noise classification.…”

Section: B Page Noise Reductionmentioning

confidence: 99%

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

Gong

Zhang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Usually, in addition to the main content, web pages contain additional information in the form of noise, such as navigation elements, sidebars and advertisements. This kind of noise has nothing to do with the main content, it will affect the tasks of data mining and information retrieval so that the sensor will be damaged by the wrong data and interference noise. Because of the diversity of web page structure, it is a challenge to detect relevant information and noise in order to improve the true reliability of sensor networks. In this paper, we propose a visual block construction method based on page type conversion (VB-PTC). This method uses a combination of site-level noise reduction based on hashtree and page-level noise reduction based on linked clusters to eliminate noise in web articles, and it successfully converts multi-record complex pages to multi-record simple pages, effectively simplifying the rules of visual block construction. In the aspect of multi-record content extraction, according to the characteristics of different fields, we use different extraction methods, combined with regular expression, natural language processing and symbol density detection methods which greatly improves the accuracy of multi-record content extraction. VB-PTC can be effectively used for information retrieval, content extraction and page rendering tasks.

show abstract

“…The post-processing modules are re-used from the previously developed texrex software (Schäfer and Bildhauer, 2012;Schäfer, 2016b;Schäfer, 2016a). The walker documents the progression of the RW in a short and a long file format.…”

Section: Built-in Processing and Output Formatsmentioning

confidence: 99%

On Bias-free Crawling and Representative Web Corpora

Schäfer

2016

Proceedings of the 10th Web as Corpus Workshop

Self Cite

View full text Add to dashboard Cite

In this paper, I present a specialized opensource crawler that can be used to obtain bias-reduced samples from the web. First, I briefly discuss the relevance of bias-reduced web corpus sampling for corpus linguistics. Then, I summarize theoretical results that show how commonly used crawling methods obtain highly biased samples from the web. The theoretical part of the paper is followed by a description my feature-complete and stable ClaraX crawler which performs so-called Random Walks, a form of crawling that allows for bias-reduced sampling if combined with methods of post-crawl rejection sampling. Finally, results from two large crawling experiments in the German web are reported. I show that bias reduction is feasible if certain technical and practical hurdles are overcome.

show abstract

Accurate and efficient general-purpose boilerplate detection for crawled web corpora

Cited by 32 publications

References 10 publications

Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

On Bias-free Crawling and Representative Web Corpora

Contact Info

Product

Resources

About