The Perceived Similarity of Photos - A Test-Collection Based Evaluation Framework for the Content-Based Image Retrieval Algorithms1

This is the unspecified version of the paper.This version of the publication may differ from the final published version. Permanent repository link Andy MacFarlane Department of Information Science, City University London, UK Pauline Rafferty Department of Information Studies, University of Aberystwyth, Wales AbstractAlthough known item searching for music can be dealt with by searching metadata using existing text search techniques, human subjectivity and variability within the music itself make it very difficult to search for unknown items. This paper examines these problems within the context of text retrieval and music information retrieval. The focus is on ascertaining a relationship between music relevance criteria and those relating to relevance judgements in text retrieval. A data-rich collection of relevance judgements by creative professionals searching for unknown musical items to accompany moving images using real world queries is analysed. The participants in our observations are found to take a socio-cognitive approach and use a range of content and context based criteria. These criteria correlate strongly with those arising from previous text retrieval studies despite the many differences between music and text in their actual content.

show abstract

“…This view is supported in image retrieval, for example, where Sormunen and colleagues [17] recommend:…”

Section: Evaluation Of Music Information Retrieval Systemsmentioning

confidence: 92%

Creative professional users’ musical relevance criteria

Inskip

MacFarlane

Rafferty

2010

Journal of Information Science

View full text Add to dashboard Cite

show abstract

“…Realistic multimedia collections are huge. For example, a photo archive of a press agency or newspaper contains hundreds of thousands of photos (Sormunen et al, 1999;ANP Beeld, 2004), and the Dutch broadcasting archive contains 70,000 hours of video material (NPS Klokhuis, 2004). For copyright reasons, it is difficult to obtain such large collections, especially if they are to be used by a larger community.…”

Section: Documentsmentioning

confidence: 99%

Using generative probabilistic models for multimedia retrieval

Westerveld

2005

SIGIR Forum

View full text Add to dashboard Cite

This thesis discusses information retrieval from multimedia archives, focusing on documents containing visual material. We investigate search and retrieval in collections of images and video, where video is defined as a sequence of still images. No assumptions are made with respect to the content of the documents; we concentrate on retrieval from generic, heterogeneous multimedia collections. In this research area a user's query typically consists of one or more example images and the implicit request is: "Find images similar to this one." In addition the query may contain a textual description of the information need. The research presented here addresses three issues within this area.First, we show how generative probabilistic models can be applied to multimedia retrieval. For each document in the collection a probabilistic model is built. For each of these models we then compute the probability that the query is generated from the model and the documents corresponding to the models with the highest probability are shown to the user. The assumption is that these are the most relevant documents, i.e., those with characteristics corresponding to the query characteristics. Visual information is modelled using Gaussian mixture models and information derived from language (e.g., the speech of the video soundtrack) is modelled using statistical language models.The second issue addressed is the parallel between the use of generative probabilistic models for multimedia retrieval and comparable models for text. This thesis describes how the techniques developed for language relate to the multimedia techniques presented here and how these parallels can be leveraged.Third, this thesis studies evaluation. We tested different model variants using a number of collections including the test collections of TRECVID, the international workshop series for benchmarking video retrieval. On average, language-based approaches outperform approaches based on visual information. However, for some queries visual information is important. A combination of both modalities gives the best results when searching a heterogeneous multimedia collection.Dissertation available from

show abstract

“…Sormunen, Markkula, and Järvelin (1999) worked closely with journalists in an investigation into how they judge similarity of photos. Again, within the journalist domain, Macaulay (2000) reported her work on a long‐term study of journalists in the newsroom.…”

Section: The Systemmentioning

confidence: 99%

Reflections on Mira: Interactive evaluation in information retrieval

Dunlop¹

2000

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

Evaluation in information retrieval (IR) has focussed largely on noninteractive evaluation of text retrieval systems. This is increasingly at odds with how people use modern IR systems: in highly interactive settings to access linked, multimedia information. Furthermore, this approach ignores potential improvements through better interface design. In 1996, the Commission of the European Union Information Technologies Programme funded a 3‐year working group, Mira, to discuss and advance research in the area of evaluation frameworks for interactive and multimedia IR applications. Led by Keith van Rijsbergen, Steve Draper, and myself from Glasgow University, this working group brought together many of the leading researchers in the evaluation domain from both the IR and human–computer interaction (HCI) communities. This article presents my personal view of the main lines of discussion that took place throughout Mira: importing and adapting evaluation techniques from HCI, evaluating at different levels as appropriate, evaluating against different types of relevance and the new challenges that drive the need for rethinking the old evaluation approaches. The article concludes that we need to consider more varied forms of evaluation to complement engine evaluation.

show abstract

The Perceived Similarity of Photos - A Test-Collection Based Evaluation Framework for the Content-Based Image Retrieval Algorithms1

Cited by 9 publications

References 22 publications

Creative professional users’ musical relevance criteria

Creative professional users’ musical relevance criteria

Using generative probabilistic models for multimedia retrieval

Reflections on Mira: Interactive evaluation in information retrieval

Contact Info

Product

Resources

About