Mining Novice User Activity with TRECVID Interactive Retrieval Tasks

Abstract

Abstract. This paper investigates the applicability of Informedia shot-based interface features for video retrieval in the hands of novice users, noted in past work as being too reliant on text search. The Informedia interface was redesigned to better promote the availability of additional video access mechanisms, and tested with TRECVID 2005 interactive search tasks. A transaction log analysis from 24 novice users shows a dramatic increase in the use of color search and shot-browsing mechanisms beyond traditional text search. In addition, a within-subjects study examined the employment of user activity mining to suppress shots previously seen. This strategy did not have the expected positive effect on performance. User activity mining and shot suppression did produce a broader shot space to be explored and resulted in more unique answer shots being discovered. Implications for shot suppression in video retrieval information exploration interfaces are discussed.

Establishing the utility of non-text search for news video retrieval with real world users

“…The students were motivated nearly to the same levels of "yes" shot counts as the expert, but with less precision, 74% later judged correct vs. 93% of the expert's "yes" shots. Details on student vs. expert performance are given elsewhere [2], but the point of interest here is folding in the analyst performances with the two treatments.…”

“…That aspect of the study is detailed elsewhere and found the mining strategy led to no performance differences [2].…”

“…All three strategies (query by text, image, concept) have been used to produce storyboard layouts of imagery by the Carnegie Mellon Informedia video search engine [1,2,3] and the MediaMill video search engine [13,14] for a number of years, with these systems scoring best for all of the TRECVID interactive video search evaluations since the task inception in 2002 [9]. Hence, there is evidence that the three strategies together are effective for the TRECVID search tasks, but there is a qualification.…”

“…Such use was productive and in greater agreement with the expert's interactions: Table 2 and Figure 6 in later sections summarize the interaction logs. The students performed well: the MAP for the 4 runs through the 24 topics ranged from 0.253 to 0.286, the highest MAP for TRECVID interactive search conducted by users outside of the system development teams [2].…”

“…TRECVID interactive evaluations historically have emphasized only information retrieval effectiveness measures (e.g., mean average precision -MAP), and not other measures of end user utility. The vast majority of TRECVID interactive runs have been conducted by the researchers themselves posing as users [9], with a few notable exceptions [1,2,3,6,17]. Even these exceptions, though, use students as subjects rather than real world users.…”