Evaluating usability based on multimodal information

Abstract

New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multiple information streams in usability evaluation. In the study, easy, medium and difficult versions of a game task were used to vary the levels of mental effort. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Four findings indicate the potential value of usability evaluations based on multimodal information: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different mental workload levels appeared to correlate with eye movement patterns, especially with a combined eye-hand movement measure. Third, HRV showed correlations with saccade speed. Finally, we present a new method using the ratio of eye fixations over mouse clicks to evaluate performance in more detail. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.

Robust Multimodal Cognitive Load Measurement (RMCLM)

“…The experimental results indicated that the use of static tool led to higher cognitive workload. In another experiment of a video game task (Lin and Imamiya, 2006), it was also found that pupil size increased when task difficulty changes from low level to high level.…”

Event-Related Assessment of Hypermedia-Based E-Learning Materials With an HRVBased Method That Considers Individual Differences in Users

“…Several studies have shown that an increase in mental effort causes a decrease in the midfrequency (MF, 0.07-0.15 Hz) power band of the HPV power spectrum [4,5,6,7,8,9,10,11].…”

Identifying Relationships between Physiological Measures and Evaluation Metrics for 3D Interaction Techniques

“…Complementary studies of Lin et al [19][20] added measures like pupillary response, eye tracking and HRV parameters to assess the user cognitive efforts in videogames. The results also showed the relationship between HRV parameters and game levels.…”

“…The above times, defined to complete the training and experiment stages, are based on Lin et al [17][18] [19][20] and Brogni et al [7][8] works, whereas the interval to rest between stages is based on Kim et al [16].…”

“…According to Malik et al [22], the physiological monitoring provides information about the user's physiological balance, and its measures are associated with stress. Researches in the Virtual Reality (VR) area have been using this type of measurement to assess the user's physical and mental effort on the 2D games [17][18] [19] [20] and to evaluate presence and user comfort in immersive virtual environments (VEs) [7][8] [16][23] [24]. However, there are no studies about the relationship between physiological measures and metrics focused on the evaluation of the quality of ITs.…”