A study of the use of simulated work task situations in interactive information retrieval evaluations

Abstract

Purpose – The purpose of this paper is to report a study of how the test instrument of a simulated work task situation is used in empirical evaluations of interactive information retrieval (IIR) and reported in the research literature. In particular, the author is interested to learn whether the requirements of how to employ simulated work task situations are followed, and whether these requirements call for further highlighting and refinement. Design/methodology/approach – In order to study how simulated work task situations are used, the research literature in question is identified. This is done partly via citation analysis by use of Web of Science®, and partly by systematic search of online repositories. On this basis, 67 individual publications were identified and they constitute the sample of analysis. Findings – The analysis reveals a need for clarifications of how to use simulated work task situations in IIR evaluations. In particular, with respect to the design and creation of realistic simulated work task situations. There is a lack of tailoring of the simulated work task situations to the test participants. Likewise, the requirement to include the test participants’ personal information needs is neglected. Further, there is a need to add and emphasise a requirement to depict the used simulated work task situations when reporting the IIR studies. Research limitations/implications – Insight about the use of simulated work task situations has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Originality/value – Simulated work task situations are widely used in IIR studies, and the present study is the first comprehensive study of the intended and unintended use of this test instrument since its introduction in the late 1990’s. The paper addresses the need to carefully design and tailor simulated work task situations to suit the test participants in order to obtain the intended authentic and realistic IIR under study.

Injecting realism into simulated work tasks: A case study of the book domain

“…Furthermore, by being the same for all the study participants experimental control is provided, and the search interactions are comparable across the group of study participants for the same SWTS. Borlund () formulates five requirements for ensuring realism in SWTSs: (1) tailoring the SWTS to the information envi‐ ronment and the participant group; (2) including participants' personal information needs; (3) randomizing the order of SWTs to prevent biases; (4) pilot‐testing the SWTSs; and (5) including the SWTSs when reporting on the study (Borlund, , p. 396, 407). She further specifies that well‐designed SWTSs are relatable and self‐identifiable for participants due to contextual richness and overlap in topical interest (Borlund, , p. 396).…”

“…Borlund () formulates five requirements for ensuring realism in SWTSs: (1) tailoring the SWTS to the information envi‐ ronment and the participant group; (2) including participants' personal information needs; (3) randomizing the order of SWTs to prevent biases; (4) pilot‐testing the SWTSs; and (5) including the SWTSs when reporting on the study (Borlund, , p. 396, 407). She further specifies that well‐designed SWTSs are relatable and self‐identifiable for participants due to contextual richness and overlap in topical interest (Borlund, , p. 396). Such tailoring of SWTSs requires a certain degree of homogeneity of the group of study participants as well as a relatively high level of abstraction in the SWTS wording (Borlund, , p. 397).…”

“…She further specifies that well‐designed SWTSs are relatable and self‐identifiable for participants due to contextual richness and overlap in topical interest (Borlund, , p. 396). Such tailoring of SWTSs requires a certain degree of homogeneity of the group of study participants as well as a relatively high level of abstraction in the SWTS wording (Borlund, , p. 397). Nevertheless, research shows that formulating realistic and engaging SWTSs remains challenging (Borlund, ).…”

“…Such tailoring of SWTSs requires a certain degree of homogeneity of the group of study participants as well as a relatively high level of abstraction in the SWTS wording (Borlund, , p. 397). Nevertheless, research shows that formulating realistic and engaging SWTSs remains challenging (Borlund, ).…”

“…Anybody who has conducted an IIR study using assigned search tasks (e.g., in the form of simulated work task situations (Borlund, 2000;2003)) will recognize that this is straightforward in theory, yet difficult in practice. Research shows that the majority of IIR studies fail to sufficiently tailor the simulated work task situations (SWTS), despite being rich in contextual descriptions, hence also disqualifying the study results (Borlund, 2016).…”