Multimodal behavioral and physiological signals as indicators of cognitive load

Zhou, Jianlong; Yu, Kun; Chen, Fang; Wang, Yang; Arshad, Syed Z.

doi:10.1145/3107990.3108002

Cited by 15 publications

(14 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As confirmed by the qualitative feedback, participants felt the standard interface to be too wordy, thus impacting their cognitive load and supporting their preference for VCA interface. 64,66,70 Guidelines proposed by Stewart and Travis (2002) 68 suggest that effective interfaces are designed by considering the capacity limitation of the users and reducing the short-term memory load. The increase in the cognitive load may affect the user’s satisfaction while completing different tasks.…”

Section: Discussionmentioning

confidence: 99%

“…In this study, participants reported higher workload scores using the standard interface than the VCA interface, results mirrored by the lower values on the technology acceptance and usability metrics. 70 In part, the high cognitive load related to the higher values on the NASA-TLX scale was due to the increased frustration involved in finding the information needed on the interface coupled with the lengthy explanations and unclear icon labels. The high scores on the technology acceptance and usability metrics for the VCA interface were due to the context-based guidance it provided during the data collection process, the real-time update of the information on the family tree and the more intuitive design of the interface.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population

Ponathil¹,

Ozkan²,

Bertrand³

et al. 2020

Health Informatics J

View full text Add to dashboard Cite

Critical for the early diagnosis of genetic disorders, a Family Health History (FHx) can be collected in several ways including electronic FHx tools, which aid easy editing and sharing by linking with other information management portals. The user acceptance of such systems is critical, especially among older adults experiencing motor and cognitive issues. This study investigated two types of FHx interfaces, standard and Virtual Conversational Agent (VCA), using 30 young (between 18 and 30) and 24 older participants (over 60). Workload, usability and performance data were collected. Even though participants required less time to complete three of five tasks on the standard interface, the VCA interface performed better in terms of subjective workload and usability. Additionally, 67% of the older adults preferred the VCA interface since it provided context-based guidance during the data collection process. The results from this study have implications for the use of virtual assistants in FHx and other areas of data collection.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population

Ponathil¹,

Ozkan²,

Bertrand³

et al. 2020

Health Informatics J

View full text Add to dashboard Cite

show abstract

“…The time duration (s) between start and end of a stroke (Oviatt et al, 2018;Zhou et al, 2018) min/max/mean/median/std stroke duration (SDR)…”

Section: Time-based Features Stroke Duration (Sdr)mentioning

confidence: 99%

“…The min/max/mean/median/std stroke duration in one sentence (Oviatt et al, 2018;Zhou et al, 2018) min/max/mean/median/std stroke speed (SS)…”

Section: Time-based Features Stroke Duration (Sdr)mentioning

confidence: 99%

Measuring and classifying students' cognitive load in pen‐based mobile learning using handwriting, touch gestural and eye‐tracking data

Li,

Luximon,

Zhang

et al. 2023

Brit J Educational Tech

View full text Add to dashboard Cite

Although the utilization of mobile technologies has recently emerged in various educational settings, limited research has focused on cognitive load detection in the pen‐based learning process. This research conducted two experimental studies to investigate what and how multimodal data can be used to measure and classify learners' real‐time cognitive load. The results found that it was a promising method to predict learners' cognitive load by analysing their handwriting, touch gestural and eye‐tracking data individually and conjunctively. The machine learning approach used in this research achieved a prediction accuracy of 0.86 area under the receiver operating characteristic curve (AUC) and 0.85/0.86 sensitivity/specificity by only using handwriting data, 0.93 AUC and 0.93/0.94 sensitivity/specificity by only using touch gestural data, and 0.94 AUC and 0.94/0.95 sensitivity/specificity by using both the touch gestural and eye‐tracking data. The results can contribute to the optimization of cognitive load and the development of adaptive learning systems for pen‐based mobile learning. Practitioner notesWhat is already known about this topic Pen‐based mobile learning systems allow natural ways of handwriting and gestural touching, which can facilitate learners' cognitive processes in mobile learning. Behavioural and physiological multimodal data are helpful in detecting learners' real‐time cognitive load in mobile learning. The effectiveness of behavioural and physiological multimodal data for measuring cognitive load in pen‐based mobile learning is limited investigated. What this paper adds This paper confirms the effectiveness of handwriting and touch gestural multimodal data for measuring pen‐based learning cognitive load, in terms of their stroke‐, path‐ and time‐based features. This paper explores the potential of eye‐tracking data in measuring pen‐based learning cognitive load. A combination of behavioural and physiological multimodal data is reported to increase the prediction accuracy for cognitive load measurement. Implications for practice and/or policy Practitioners are suggested to use behavioural and physiological multimodal data individually or conjunctively for measuring cognitive load in pen‐based learning. The results provide guides for developing adaptive pen‐based learning systems by optimizing the real‐time cognitive load.

show abstract

“…While the physiological modality may be impractical to record for an embodied agent in the wild, facial data, and pose data could be used, in combination, by any agent that has video cameras. Zhou et al (2018) present a case study on the measurement of cognitive load through multimodal physiological features connected to a computer. While the output of the systems described by Zhou et al-an estimation of cognitive loadwere not in themselves interesting for the purposed described in this paper, the work presents a good example of how the types of sensors mentioned by Marechal et al can be used in a computer interface, how they can be multimodally fused (with beneficial results presented by Zhou et al), and how intrusive such an interface is.…”

Section: Body Pose and Facial Expressionsmentioning

confidence: 99%

Modeling Feedback in Interaction With Conversational Agents—A Review

Axelsson

Buschmeier

Skantze

2022

Front. Comput. Sci.

View full text Add to dashboard Cite

Intelligent agents interacting with humans through conversation (such as a robot, embodied conversational agent, or chatbot) need to receive feedback from the human to make sure that its communicative acts have the intended consequences. At the same time, the human interacting with the agent will also seek feedback, in order to ensure that her communicative acts have the intended consequences. In this review article, we give an overview of past and current research on how intelligent agents should be able to both give meaningful feedback toward humans, as well as understanding feedback given by the users. The review covers feedback across different modalities (e.g., speech, head gestures, gaze, and facial expression), different forms of feedback (e.g., backchannels, clarification requests), and models for allowing the agent to assess the user's level of understanding and adapt its behavior accordingly. Finally, we analyse some shortcomings of current approaches to modeling feedback, and identify important directions for future research.

show abstract

Multimodal behavioral and physiological signals as indicators of cognitive load

Cited by 15 publications

References 92 publications

An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population

An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population

Measuring and classifying students' cognitive load in pen‐based mobile learning using handwriting, touch gestural and eye‐tracking data

Modeling Feedback in Interaction With Conversational Agents—A Review

Contact Info

Product

Resources

About