Effects of granularity of search results on the relevance judgment behavior of engineers: Building systems for retrieval and understanding of context

Balatsoukas, Panos; Demian, Peter

doi:10.1002/asi.21268

Cited by 4 publications

(4 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While it is difficult to estimate relevance accurately, the complexity of the problem is multiplied when asking judges to estimate relevance of objects at multiple levels of granularity. In [4], engineers were asked to assess relevance of large documents containing structured data at multiple levels. For evaluating our search systems, we adapt their relevance scale [17,20] designed specifically for assessing relevance at multiple granularity.…”

Section: Graded Relevance Scalementioning

confidence: 99%

One size does not fit all

Ganu

Marian

2013

Proceedings of the 22nd ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Users rely increasingly on online forums, blogs, and mailing lists to exchange information, practical tips, and stories. Although this type of social interaction has become central to our daily lives and decision-making processes, forums are surprisingly technologically poor: often there is no choice but to browse through massive numbers of posts while looking for specific information. A critical challenge then for forum search is to provide results that are as complete as possible and that do not miss some relevant information but that are not too broad. In this paper, we address the problem of presenting textual search results in a concise manner to answer user needs. Specifically, we propose a new search approach over free-form text in forums that allows for the search results to be returned at varying granularity levels. We implement a novel hierarchical representation and scoring technique for objects at multiple granularities, taking into account the inherent containment relationship provided by the hierarchy. We also present a score optimization algorithm that efficiently chooses the best k-sized result set while ensuring no overlap between the results. We evaluate the effectiveness of multi-granularity search by conducting extensive user studies and show that a mixed granularity set of results is more relevant to users than standard post-only approaches.

show abstract

Section: Graded Relevance Scalementioning

confidence: 99%

One size does not fit all

Ganu

Marian

2013

Proceedings of the 22nd ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

show abstract

“…A large number of studies were conducted on the evaluation of interactive information retrieval (e.g., Balatsoukas & Demian, 2010;Joho, Birbeck, & Jose, 2007;Joho & Jose, 2008;Jose, Furner, & Harper, 1998;Pharo, 2008;Urban & Jose, 2005 1994199519961997199820002002200420062011Fig. 1.…”

Section: Experimental Studiesmentioning

confidence: 99%

Scenarios in information seeking and information retrieval research: A methodological application and discussion

Kim

2012

Library & Information Science Research

View full text Add to dashboard Cite

“…In most of these studies, a collection of scientific articles in the domain of informatics (IEEE) were used. A notable exception from the INEXgenerated studies is the study by Balatsoukas and Demian (2010) on XML-coded documentation. It is reasonable to believe that genre will influence the way users read a document, which, in turn, might influence which element types are most appropriate for satisfying an information need.…”

Section: Introductionmentioning

confidence: 99%

The effect of task type on preferred element types in an XML-based retrieval system

Pharo

Krahn

2011

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

This article examines the influence of task type on the users' preferred level of document elements (full articles, sections, or subsections) during interaction with an XMLversion ofWikipedia.We found that in general articles and subsections seemed to be the most valuable elements for our test subjects. For information-gathering tasks, this tendency was stronger, whereas for fact-finding tasks, the sections seemed to play a more important role. We assume from this that users select different information search strategies for the two task types. When dealing with fact-finding tasks, users seem more likely to use one single element as an answer, while when they do information gathering, they pick information from several elements. IntroductionAlthough many information retrieval (IR) systems index and retrieve full documents, they seldom index the entire document and its parts independently. Nevertheless, in many situations, only parts of documents will be relevant to a user's information need. In this article, we report from a study of users' preferences with respect to document parts and how these differ for two types of tasks.XML offers the possibility of indexing and retrieving semantically meaningful document parts (Luk et al., 2002). An essential question is what types of elements in an XML-hierarchy are the most useful for users? In most XMLdocuments, the markup is quite detailed; it is, however, not likely that users are interested in being presented with very small elements such as titles or links. In the context of the Initiative for the Evaluation of XML retrieval (INEX), several authors have studied which element types users prefer when searching in XML retrieval systems a collection of scientific articles in the domain of informatics (IEEE) were used. A notable exception from the INEXgenerated studies is the study by Balatsoukas and Demian (2010) on XML-coded documentation. It is reasonable to believe that genre will influence the way users read a document, which, in turn, might influence which element types are most appropriate for satisfying an information need. A second influencing factor might be the type of work task for which the information is needed (Byström & Järvelin, 1995), whereas users' topic knowledge (Marchionini, 1995) could be a third factor.In this study, we have investigated test subjects searching an XML-tagged version of Wikipedia. The data were collected in 2007 for the INEX Interactive Track. The search tasks analyzed in this study were categorized as two different types: information gathering and fact-finding. As part of the experiment, the test subjects were asked to fill in a questionnaire on, among other things, their knowledge about the topic, but a preliminary analysis of this data set made it impossible to draw any conclusions. The research questions examined in this study are as follows:• What element types do people use when searching in an XMLversion of Wikipedia? • How does the task type influence users element-type preferences?We begin presenting related studies and th...

show abstract

Effects of granularity of search results on the relevance judgment behavior of engineers: Building systems for retrieval and understanding of context

Cited by 4 publications

References 45 publications

One size does not fit all

One size does not fit all

Scenarios in information seeking and information retrieval research: A methodological application and discussion

The effect of task type on preferred element types in an XML-based retrieval system

Contact Info

Product

Resources

About