Turn the Page: Automated Traversal of Paginated Websites

Abstract

Content-intensive web sites, such as Google or Amazon, paginate their results to accommodate limited screen sizes. Thus, human users and automatic tools alike have to traverse the pagination links when they crawl the site, extract data, or automate common tasks, where these applications require access to the entire result set. Previous approaches, as well as existing crawlers and automation tools, rely on simple heuristics (e.g., considering only the link text), falling back to an exhaustive exploration of the site where those heuristics fail. In particular, focused crawlers and data extraction systems target only fractions of the individual pages of a given site, rendering a highly accurate identification of pagination links essential to avoid the exhaustive exploration of irrelevant pages. We identify pagination links in a wide range of domains and sites with near perfect accuracy (99%). We obtain these results with a novel framework for web block classification, BER y L, that combines rule-based reasoning for feature extraction and machine learning for feature selection and classification. Through this combination, BER y L is applicable in a wide settings range, adjusted to maximise either precision, recall, or speed. We illustrate how BER y L minimises the effort for feature extraction and evaluate the impact of a broad range of features (content, structural, and visual). The research leading to these results has received funding from the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) / ERC grant agreement DIADEM, no. 246858.

A usability evaluation of Web user interface scrolling types

“…Most Web sites use pagination to present a set number of results in Web search results or demonstrating a set number of posts while seeing a discussion string. Pagination can be used as a part of some structure on each Web application to isolate returned information and show it on different pages (Furche, et al, 2012). Pagination additionally incorporates a rationale of planning and showing connections to different pages.…”

The Augmented Web

“…Location maintenance is addressed through both preventive and curative approaches (see Grace et al [2011] for an overview). The former attempts to avoid locator failure by providing more robust XPath expressions [Kowalkiewicz et al 2006;Paz and Díaz 2010].…”

Data Science with Vadalog: Bridging Machine Learning and Reasoning

“…Consider the problem of identifying semantic blocks on a web page, such as pagination bars, navigation menus, headers, footers, and sidebars [26,35]. The page is represented by the DOM tree and CSS model.…”