On Finding Templates on Web Collections

Abstract

Templates are pieces of HTML code common to a set of web pages usually adopted by content providers to enhance the uniformity of layout and navigation of theirs Web sites. They are usually generated using authoring/publishing tools or by programs that build HTML pages to publish content from a database. In spite of their usefulness, the content of templates can negatively affect the quality of results produced by systems that automatically process information available in web sites, such as search engines, clustering and automatic categorization programs. Further, the information available in templates is redundant and thus processing and storing such information just once for a set of pages may save computational resources. In this paper, we present and evaluate methods for detecting templates considering a scenario where multiple templates can be found in a collection of Web pages. Most of previous work have studied template detection algorithms in a scenario where the collection has just a single template. The scenario with multiple templates is more realistic and, as it is discussed here, it raises important questions that may require 172 World Wide Web (2009) 12:171-211 extensions and adjustments in previously proposed template detection algorithms. We show how to apply and evaluate two template detection algorithms in this scenario, creating solutions for detecting multiple templates. The methods studied partitions the input collection into clusters that contain common HTML paths and share a high number of HTML nodes and then apply a single-template detection procedure over each cluster. We also propose a new algorithm for single template detection based on a restricted form of bottom-up tree-mapping that requires only small set of pages to correctly identify a template and which has a worst-case linear complexity. Our experimental results over a representative set of Web pages show that our approach is efficient and scalable while obtaining accurate results.

Impact of template removal on Web search DOI 10.5752/P.2316-9451.2012v1n1p28

“…Despite the general belief that template removal may improve the quality of search results (VIEIRA;PINTO, 2006;YI;LIU;LI, 2003), our experiments suggest the improvements are not consistent across various sites, and that template removal in general does not cause improvements on quality of results provided by search systems. Indeed, in three of the test collections used in the intrasite scenario, the results obtained with template removal were equivalent to results without template removal, for the queries we experimented with.…”

“…Many authors have hypothesized that templates are typically not related to the main content of the pages and might hurt search quality (BARYOSSEF; RAJAGOPALAN, 2002;VIEIRA;PINTO, 2006;YI;LIU;LI, 2003). Such an idea has motivated the proposal of algorithms to detect and remove templates which led to gains according to preliminary evaluations (BARYOSSEF; RAJAGOPALAN, 2002; VIEIRA; PINTO, 2006; YI; LIU; LI, 2003).…”

TeMex

“…This was computed by calculating a CS of size 3. We observed that other techniques such as [4,10,7,9,11] obtain good values of F1 in certain webpages, but they are manually feed with collections of webpages that share the same template. With this conditions, our tool produces an F1 close to 95% in most of the cases.…”

“…Some of them measure the number of words correctly retrieved [10,9]. This can be rather imprecise, because it ignores the structure (e.g., div, table...) retrieved.…”

Assessing the Effort of Repairing the Accessibility of Web Sites

“…Towards information retrieval, template detection can positively impact performance and resource usage in processes of analysis of HTML pages [5]. Regarding Web page Clustering, templates could help in cluster structurally similar Web pages [4].…”