Metadata harvesting for content‐based distributed information retrieval

Abstract

We propose an approach to content-based Distributed Information Retrieval based on the periodic and incremental centralization of full-content indices of widely dispersed and autonomously managed document sources. Inspired by the success of the Open Archive Initiative's (OAI) Protocol for metadata harvesting, the approach occupies middle ground between content crawling and distributed retrieval. As in crawling, some data move toward the retrieval process, but it is statistics about the content rather than content itself; this grants more efficient use of network resources and wider scope of application. As in distributed retrieval, some processing is distributed along with the data, but it is indexing rather than retrieval; this reduces the costs of content provision while promoting the simplicity, effectiveness, and responsiveness of retrieval. Overall, we argue that the approach retains the good properties of centralized retrieval without renouncing to costeffective, large-scale resource pooling. We discuss the requirements associated with the approach and identify two strategies to deploy it on top of the OAI infrastructure. In particular, we define a minimal extension of the OAI protocol which supports the coordinated harvesting of full-content indices and descriptive metadata for content resources. Finally, we report on the implementation of a proof-of-concept prototype service for multimodel content-based retrieval of distributed file collections. IntroductionOur interest is in content-based retrieval of widely dispersed and autonomously managed document sources. 1 This is the central problem of Distributed Information Retrieval (DIR), and over the past 10 years, it has been mainly approached by distributing the retrieval process along with the data: Queries have been "pushed" toward the content, and the results of their local execution have been centrally gathered and presented to the user (cf. Callan, 2000a).Traditionally, distributed retrieval services have relied on simple client/server architectures in which brokers route queries submitted by local or remote clients toward a number of mutually autonomous and potentially uncooperative retrieval engines. Figure 1 shows how client/server distributed retrieval works. A Search Broker B interfaces Clients C and dispatches their Queries Q to a number of autonomous search engines. S 1 , S 2 , S n , each of which executes it against an Index FT i of some Content C i before returning Results R i back to B, which merges them and relays them to C. Optionally, B optimizes query distribution by selecting a subset of the engines based on previously gathered descriptions of their content. Based on summary descriptions of the content served by each engine, advanced techniques of source selection and data fusion have been produced to, respectively, minimize network In the lack of a well-established terminology, we use the term contentbased to characterize retrieval processes defined over indices of essentially unstructured documents. Content-based retrieval lies at one...

Query‐performance prediction for effective query routing in domain‐specific repositories

“…In addition to the practical implications of this work, the findings on the characteristics of domain-specific repositories, improvements in query-performance prediction, and application of query-performance prediction for improving retrieval performance in distributed repositories can have implications for several IR-research areas including the design of domain-specific repositories (Bhavnani et al, 2006;Finn & Kushmerick, 2006;Marcial & Hemminger, 2010;Muresan & Klavans, 2013;Pattuelli, 2011;Tang, Yang, & Song, 2013), distributed repositories and federated search (Avrahami et al, 2006;Davis & Lagoze, 2000; Paltoglou et al, 2011;Simeoni et al, 2008), query expansion (Alemayehu, 2003;Efthimiadis, 2000;Minker et al, 1973;Shiri & Revie, 2006), query analysis and characteristics (Bashir & Rauber, 2011), and proactive documentrecommendation systems (Liu et al, 2012).…”

“…The improvements in query performance prediction can lead to several new applications for improving IR including selective query expansion (Alemayehu, 2003;Efthimiadis, 2000;Minker, Peltola, & Wilson, 1973;Shiri & Revie, 2006), federated search and query routing (Avrahami, Yau, Si, & Callan, 2006;Davis & Lagoze, 2000;Paltoglou, Salampasis, & Satratzemi, 2011;Simeoni, Yakici, Neely, & Crestani, 2008) and query-based proactive document recommendation systems (Liu, Lai, & Chen, 2012). For example, users can be recommended query-expansion terms for improved retrieval based on the predicted performance of alternative expanded queries.…”

“…SVM = support vector machine; CISI = Information Science Institute Test Collection; CRAN = Comprehensive R Archive Network; TIME = TIME Magazine; TREC9 = Ninth Text Retrieval Conference; CACM = Communications of the ACM. Paltoglou et al, 2011;Simeoni et al, 2008), query expansion (Alemayehu, 2003;Efthimiadis, 2000;Minker et al, 1973;Shiri & Revie, 2006), query analysis and characteristics (Bashir & Rauber, 2011), and proactive documentrecommendation systems (Liu et al, 2012).…”

Beyond Theory: Preparing Dublin Core Metadata for OAI-PMH Harvesting

“…Works about the harvesting process itself have tended either to be quite technical (Simeoni, Yakici, Neely, & Crestani, 2008) or to offer a general view of how a metadata harvesting process works in relation to issues of metadata quality and analysis of Dublin Core usage (Elings & Waibel, 2007;Graham, 2001;Hagedorn, 2003;Ward, 2004). Cole and Foulonneau's Using the Open Archives Initiative for Metadata Harvesting (2007) covers both the technical operations of OAI-PMH and how it works on the ground for those out there creating and managing metadata in their local environments.…”

Developing a Metadata Best Practices Model: The Experience of the Colorado State University Libraries

“…Medeiros (2006), Shreeves, Riley, and Milewicz (2006), and Elings and Waibel (2007) all described the importance of metadata sharing and discussed issues with metadata aggregation. Simeoni, Yakici, Neely, and Crestani (2008) focused on a content-based distributed information retrieval approach which, they stated, "occupies middle ground between content crawling and distributed retrieval" (p. 12).…”