Sampling strategies for information extraction over the deep web

Abstract

Information extraction systems discover structured information in natural language text. Having information in structured form enables much richer querying and data mining than possible over the natural language text. However, information extraction is a computationally expensive task, and hence improving the efficiency of the extraction process over large text collections is of critical interest. In this paper, we focus on an especially valuable family of text collections, namely, the so-called deep-web text collections, whose contents are not crawlable and are only available via querying. Important steps for efficient information extraction over deep-web text collections (e.g., selecting the collections on which to focus the extraction effort, based on their contents; or learning which documents within these collections-and in which order-to process, based on their words and phrases) require having a representative document sample from each collection. These document samples have to be collected by querying the deepweb text collections, an expensive process that renders impractical the existing sampling approaches developed for other data scenarios. In this paper, we systematically study the space of query-based document sampling techniques for information extraction over the deep web. Specifically, we consider (i) alternative query execution schedules, which vary on how they account for the query effectiveness, and (ii) alternative document retrieval and processing schedules, which vary on how they distribute the extraction effort over documents. We report the results of the first large-scale experimental evaluation of sampling techniques for information extraction over the deep web. Our results show the merits and limitations of the alternative query execution and document retrieval and processing strategies, and provide a roadmap for addressing this critically important building block for efficient, scalable information extraction.

Design and implementation of crawling algorithm to collect deep web information for web archiving

“…Even though previous studies found it difficult to perform deep web collection, they require multiple keywords in order to gather associated information if available (Barrio and Gravano, 2017). Keywords should be managed according to related themes or ranges.…”

Ranked Deep Web Page Detection Using Reinforcement Learning and Query Optimization

“…M is the modeling function, gt isthegroundtruth, s isthemodifiedKendall'sTauscore,and S istheModified Kendall'sTauscoringfunction.Duringlearning,scoreswerebackpropagatedinacrawlgraphwith thehelpofreinforcementlearning.FourdatasetsofDefenseAdvancedResearchProjectsAgency (DARPA)Memexprojectwereused.Itachievedtherelevancyscoreofvalue0.698. Barrioetal.recommendedasystematicquery-basedtechniqueusedforbuildingahigh-quality document sample (Barrio & Gravano, 2017). A representative sample is always needed, which representsthedeepwebcollection.Thistechniquewasbasedonthequeryexecutionorder,revision ofqueryorder,andfiltrationofqueries.Opendirectoryprojectwasusedasadatasetthatcontains 335realwebcollections.Performancemetricssuchascoverage,samplesize,uniquetuples,and issuequerieswereusedtoevaluatethesample-basedtechnique.Improvedcoverageandsampling efficiencywerereportedratherthanexactvalues.Thisquery-basedtechniquecanbeusedinfocused crawlingforthedeepweb.…”

Web based Content Extraction and Retrieval in Web Engineering

“…The research concentrates on a particularly respected family of text groups, individually, the so-called deep-web text groups, whose insides aren't crawl-able and are merely obtainable through enquiring. There is a very significant step for effective material extraction over deep-web text groups [22].Wang and Stewart 2015 studied geographic information science, modeling geographic dynamics found on spatiotemporal material mined from a Web, particularly unconstructed facts like online news reports. Consideration of spatiotemporal besides semantic data from a group of Web forms allows us to shape a rich exemplification of geographic details labeled in the text, taking where, when, or what proceedings have happened.…”