A DOM-structural Cohesion Analysis Approach for Segmentation of Modern Web Pages

Abstract

Web page segmentation is a fundamental technique applied in information retrieval systems to enhance web crawling tasks and information extraction. Its purpose is to gain deep insights from crawling results and extract the main content of a webpage by disregarding the irrelevant regions. Over time, several solutions have been proposed to address the segmentation problem using different approaches and learning strategies. Among these, the structural cue, which is a characteristic of the DOM tree, is widely utilized as a primary factor in segmentation models. In this paper, we propose a novel technique for web page segmentation using DOM-structural cohesion analysis. Our approach involves generating blocks that represent groups of DOM subtrees with similar tag structures. By analyzing the cohesion within each generated block and comparing detailed information such as types, attributes, and visual cues of web page elements, we can effectively maintain or reconstruct the segmentation layout. Additionally, we employ the Canny algorithm to optimize the segmentation result by reducing redundant spaces, resulting in a more correct segmentation. We evaluate the effectiveness of our approach using a dataset of 1,969 web pages. The approach achieves 64% on the FB3 score, surpassing existing state-of-the-art methods. The proposed DOM-structural cohesion analysis has the potential for improving web page segmentation and its various applications.