Site-Level Web Template Extraction Based on DOM Analysis

Abstract

Abstract. One of the main development resources for website engineers are Web templates. Templates allow them to increase productivity by plugin content into already formatted and prepared pagelets. For the final user templates are also useful, because they provide uniformity and a common look and feel for all webpages. However, from the point of view of crawlers and indexers, templates are an important problem, because templates usually contain irrelevant information such as advertisements, menus, and banners. Processing and storing this information leads to a waste of resources (storage space, bandwidth, etc.). It has been measured that templates represent between 40% and 50% of data on the Web. Therefore, identifying templates is essential for indexing tasks. In this work we propose a novel method for automatic web template extraction that is based on similarity analysis between the DOM trees of a collection of webpages that are detected using an hyperlink analysis.Our implementation and experiments demonstrate the usefulness of the technique.

What Web Template Extractor Should I Use? A Benchmarking and Comparison for Five Template Extractors

“…• Limitations/Problems: The main limitation is that the evaluation was done with only 10 websites, and also the 24 webpages used were not randomly selected, but all of them implementing the template (this scenario is easier for a template extractor). TemEx (2015) [2]: As in RTDM-TD and RBM-TD, this algorithm also uses a mapping between the DOM trees to determine what nodes belong to all webpages, but it does not force a node to belong to all webpages, only to a subset, as in SST. In this respect, this algorithm is more democratic, because it uses a number of votes to determine that a node has been repeated in enough webpages to be considered as part of the template.…”

“…The reason is that each technique has been implemented with a different language and with different components that affect the efficiency. But also, they have been evaluated with different evaluation criteria (e.g., counting retrieved words [53,54] vs. characters [25] vs. DOM nodes [2] vs. text blocks [47,55]) and with a different collection of benchmarks. Using different benchmarks to compare template extractors is unacceptable because some techniques used artificial [6] (automatically generated webpages sharing exactly the same template) and others used real webpages implemented by heterogenous designers [2,54].…”

“…But also, they have been evaluated with different evaluation criteria (e.g., counting retrieved words [53,54] vs. characters [25] vs. DOM nodes [2] vs. text blocks [47,55]) and with a different collection of benchmarks. Using different benchmarks to compare template extractors is unacceptable because some techniques used artificial [6] (automatically generated webpages sharing exactly the same template) and others used real webpages implemented by heterogenous designers [2,54]. Similarly, some used random webpages [26,56,57], possibly using different templates, while others manually provided webpages that implement exactly the same template [53,54].…”

“…To solve this, we reimplemented from scratch the main template extraction systems in the literature [2,53,54,56,57]. All of them are now available as open-source at: http://www.dsic.upv.…”

Information Retrieval Based on DOM Trees

“…. 6.1 Web page of www.lemonde.fr's website and its main content (extracted with our web content extraction tool) . .…”