HEMVIP: Human Evaluation of Multiple Videos in Parallel

Abstract

In many research areas, for example motion and gesture generation, objective measures alone do not provide an accurate impression of key stimulus traits such as perceived quality or appropriateness. The gold standard is instead to evaluate these aspects through user studies, especially subjective evaluations of video stimuli. Common evaluation paradigms either present individual stimuli to be scored on Likert-type scales, or ask users to compare and rate videos in a pairwise fashion. However, the time and resources required for such evaluations scale poorly as the number of conditions to be compared increases. Building on standards used for evaluating the quality of multimedia codecs, this paper instead introduces a framework for granular rating of multiple comparable videos in parallel.This methodology essentially analyses all condition pairs at once. Our contributions are 1) a proposed framework, called HEMVIP, for parallel and granular evaluation of multiple video stimuli and 2) a validation study confirming that results obtained using the tool are in close agreement with results of prior studies using conventional multiple pairwise comparisons.CCS Concepts: • Human-centered computing → Human computer interaction (HCI).

A Large, Crowdsourced Evaluation of Gesture Generation Systems on Common Data: The GENEA Challenge 2020

“…Although there are similarities, the two orderings are meaningfully different. This, together with the results in [25], reinforces a conclusion that the two studies managed to disentangle aspects of perceived motion quality (human-likeness) from the perceived link between gesture and speech (appropriateness). Figure 5, meanwhile, visualises confidence regions for the median rating as boxes whose horizontal and vertical extents are 2: Box plots visualising the ratings distribution in the two studies.…”

“…These speech segments, which were not revealed to participants, were selected across the test inputs to be full and/or coherent phrases. The motion from the corresponding intervals in the BVH files submitted by participating teams was extracted and converted to a motion video originates from [25], and was changed for each of the two evaluations in this paper. clip using the visualisation server provided to participants (see Section 5.1), albeit at a higher resolution of 960×540 this time.…”

“…For a detailed explanation of the evaluation interface we refer the reader to [25], which introduced and validated the evaluation paradigm for gesture-motion stimuli.…”

“…The question asked in the image ("How well do the character's movements reflect what the character says? ")originates from[25], and was changed for each of the two evaluations in this paper.…”

SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents

“…Each evaluation page presented the videos of four conditions for the same speech, and a participant rated each video on a scale of 0-100. This evaluation method [17] was inspired by MUSHRA [42], which is the standardized evaluation method for comparing audio qualities. We evaluated two different aspects of gestures as the GENEA Challenge did.…”