Finding seeds to bootstrap focused crawlers

Vieira, Karane; Barbosa, Luciano; Silva, Altigran Soares; Freire, Juliana; Moura, Edleno Silva de

doi:10.1007/s11280-015-0331-7

Cited by 22 publications

(21 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al. [31] proposed a system that uses relevance feedback to gather seeds to bootstrap focused crawlers. It submits keyword search queries to Bing; extracts keywords from the result pages classified as relevant for the focus domain; and uses these keywords to construct new search queries.…”

Section: Search-based Discoverymentioning

confidence: 99%

“…Several focused crawling and discovery techniques could potentially be adapted for this problem. However, they all rely on the availability of an accurate domain-specific classifier [3,13,31]. This is an unrealistic assumption for the many application scenarios where experts must start with a small set of relevant websites, since a small sample is unlikely to be sufficient construct an accurate classifier.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bootstrapping Domain-Specific Content Discovery on the Web

Pham

Santos

Freire

2019

The World Wide Web Conference

Self Cite

View full text Add to dashboard Cite

The ability to continuously discover domain-specific content from the Web is critical for many applications. While focused crawling strategies have been shown to be effective for discovery, configuring a focused crawler is difficult and time-consuming. Given a domain of interest D, subject-matter experts (SMEs) must search for relevant websites and collect a set of representative Web pages to serve as training examples for creating a classifier that recognizes pages in D, as well as a set of pages to seed the crawl. In this paper, we propose DISCO, an approach designed to bootstrap domain-specific search. Given a small set of websites, DISCO aims to discover a large collection of relevant websites. DISCO uses a ranking-based framework that mimics the way users search for information on the Web: it iteratively discovers new pages, distills, and ranks them. It also applies multiple discovery strategies, including keywordbased and related queries issued to search engines, backward and forward crawling. By systematically combining these strategies, DISCO is able to attain high harvest rates and coverage for a variety of domains. We perform extensive experiments in four social-good domains, using data gathered by SMEs in the respective domains, and show that our approach is effective and outperforms state-ofthe-art methods. CCS CONCEPTS• Information systems → Web searching and information discovery; Page and site ranking; Content ranking; Rank aggregation.

show abstract

Section: Search-based Discoverymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Bootstrapping Domain-Specific Content Discovery on the Web

Pham

Santos

Freire

2019

The World Wide Web Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rather, most existing unsupervised approaches [2,8,9,29,30,34,35] can be applied either over a collection of result pages, or over a collection of detail pages. Other approaches rely on the same publishing pattern, but focus only on segmenting the result pages [28], or rely on the much weaker signals arising from the aligning of the labels of the fields on the search form directly against the labels of the data on the detail pages [37]; finally, there are several approaches that focus on the problem of finding redundancy [3,38] among several sites, but the problem quickly trespasses on that of integrating data coming from autonomous sources [3,5,6,23,38], a problem that is well known not to have a simple solution [14].…”

Section: Introduction and Overviewmentioning

confidence: 99%

“…The following related problems have been already tackled in literature and are beyond the scope of the present paper: finding deep Web sources [36,39,40,43]; filling the search fields with meaningful values to collect result pages [1,4,17,25,[31][32][33]42]; crawling paginated search result pages [21].…”

Section: Introduction and Overviewmentioning

confidence: 99%

RED: Redundancy-Driven Data Extraction from Result Pages?

Guo¹,

Crescenzi

Furche

et al. 2019

The World Wide Web Conference

View full text Add to dashboard Cite

Data-driven websites are mostly accessed through search interfaces. Such sites follow a common publishing pattern that, surprisingly, has not been fully exploited for unsupervised data extraction yet: the result of a search is presented as a paginated list of result records. Each result record contains the main attributes about one single object, and links to a page dedicated to the details of that object. We present red, an automatic approach and a prototype system for extracting data records from sites following this publishing pattern. red leverages the inherent redundancy between result records and corresponding detail pages to design an effective, yet fully-unsupervised and domain-independent method. It is able to extract from result pages all the attributes of the objects that appear both in the result records and in the corresponding detail pages. With respect to previous unsupervised methods, our method does not require any a priori domain-dependent knowledge (e.g, an ontology), can achieve a significantly higher accuracy while automatically selecting only object attributes, a task which is out of the scope of traditional fully unsupervised approaches. With respect to previous supervised or semi-supervised methods, red can reach similar accuracy in many domains (e.g., job postings) without requiring supervision for each domain, let alone each website. CCS CONCEPTS • Information systems → Data extraction and integration.

show abstract

“…To direct the crawl towards topically relevant websites (i.e., websites with content relevant to cultural heritage) we use an SVM classifier, which is trained by resorting to an equal number of positive and negative examples of websites that are used as input to the model builder component of ACHE. Subsequently, the seed finder [79] component is used to aid the process of locating initial seeds for the focused crawl on the clear web; this is achieved by combining the pre-built classification model with the topic-related user-provided query discussed above. Since the crawled websites may be anything from blog posts to organizational web pages and do not have a predetermined structure (unlike the social media pages), the collected content is only parsed to remove HTML markup and is stored as raw text in the Hydria data lake.…”

mentioning

confidence: 99%

Hydria: An Online Data Lake for Multi-Faceted Analytics in the Cultural Heritage Domain

Deligiannis

Raftopoulou

Tryfonopoulos

et al. 2020

BDCC

View full text Add to dashboard Cite

Advancements in cultural informatics have significantly influenced the way we perceive, analyze, communicate and understand culture. New data sources, such as social media, digitized cultural content, and Internet of Things (IoT) devices, have allowed us to enrich and customize the cultural experience, but at the same time have created an avalanche of new data that needs to be stored and appropriately managed in order to be of value. Although data management plays a central role in driving forward the cultural heritage domain, the solutions applied so far are fragmented, physically distributed, require specialized IT knowledge to deploy, and entail significant IT experience to operate even for trivial tasks. In this work, we present Hydria, an online data lake that allows users without any IT background to harvest, store, organize, analyze and share heterogeneous, multi-faceted cultural heritage data. Hydria provides a zero-administration, zero-cost, integrated framework that enables researchers, museum curators and other stakeholders within the cultural heritage domain to easily (i) deploy data acquisition services (like social media scrapers, focused web crawlers, dataset imports, questionnaire forms), (ii) design and manage versatile customizable data stores, (iii) share whole datasets or horizontal/vertical data shards with other stakeholders, (iv) search, filter and analyze data via an expressive yet simple-to-use graphical query engine and visualization tools, and (v) perform user management and access control operations on the stored data. To the best of our knowledge, this is the first solution in the literature that focuses on collecting, managing, analyzing, and sharing diverse, multi-faceted data in the cultural heritage domain and targets users without an IT background.

show abstract

Finding seeds to bootstrap focused crawlers

Cited by 22 publications

References 21 publications

Bootstrapping Domain-Specific Content Discovery on the Web

Bootstrapping Domain-Specific Content Discovery on the Web

RED: Redundancy-Driven Data Extraction from Result Pages?

Hydria: An Online Data Lake for Multi-Faceted Analytics in the Cultural Heritage Domain

Contact Info

Product

Resources

About