TEXT: Automatic Template Extraction from Heterogeneous Web Pages

Abstract

World Wide Web is the most useful source of information. In order to achieve high productivity of publishing, the webpages in many websites are automatically populated by using the common templates with contents. The templates provide readers easy access to the contents guided by consistent structures. However, for machines, the templates are considered harmful since they degrade the accuracy and performance of web applications due to irrelevant terms in templates. Thus, template detection techniques have received a lot of attention recently to improve the performance of search engines, clustering, and classification of web documents. In this paper, we present novel algorithms for extracting templates from a large number of web documents which are generated from heterogeneous templates. We cluster the web documents based on the similarity of underlying template structures in the documents so that the template for each cluster is extracted simultaneously. We develop a novel goodness measure with its fast approximation for clustering and provide comprehensive analysis of our algorithm. Our experimental results with real-life data sets confirm the effectiveness and robustness of our algorithm compared to the state of the art for template detection algorithms.

Authoring of Personalized Web Page from Heterogeneous Web Pages by Content Extraction and Integration

“…So, different values are assigned to each tag and the total value is calculated which is used to decide whether a block is noise [3]. C. Kim and K. Shim propose a novel algorithm to extract template from numerous documents constructed by heterogeneous template [4].…”

PHP-sensor

“…Now, assume a web application"s page w ∈ W, we categorize it into some of likely template (L) and recover various output parameters (O) consequently. Procedure for drawing out patterns of web page or template embedded in web page has been obtainable in current state-of-techniques [13,14]. Although, we utilize some of the techniques from TEXT [13] that highlights the DOM tree of a page reflecting the required paths.…”

“…Procedure for drawing out patterns of web page or template embedded in web page has been obtainable in current state-of-techniques [13,14]. Although, we utilize some of the techniques from TEXT [13] that highlights the DOM tree of a page reflecting the required paths. Our web page pattern mining technique comprises subsequent four stages.…”

Template Extraction from Heterogeneous Web Pages with Cosine Similarity

“…Note in the previous model this search space is equal to all other clusters in set C. Now we can start to find the first best merging pair in GetInitBestPair method. To calculate the MDL cost we have to calculate the number of 1's in MT,-1 & +1 in M∆ according to Lemma3 [8]. And finally we can merge the two clusters whose MDL cost is the minimum one.…”

“…To simplify the clustering calculation the web document is represented in DOM tree format [5]. After that essential paths of this web document is extracted and represented into the ME matrix [8]. Each column & row in ME represents the document and the essential paths of that document respectively.…”