FATHOM: A neural network-based non-verbal human comprehension detection system for learning environments

Abstract

This paper presents the application of FATHOM, a computerised non-verbal comprehension detection system, to distinguish participant comprehension levels in an interactive tutorial. FATHOM detects high and low levels of human comprehension by concurrently tracking multiple non-verbal behaviours using artificial neural networks. Presently, human comprehension is predominantly monitored from written and spoken language. Therefore, a large niche exists for exploring human comprehension detection from a non-verbal behavioral perspective using artificially intelligent computational models such as neural networks. In this paper, FATHOM was applied to a video-recorded exploratory study containing a learning task designed to elicit high and low comprehension states from the learner. The learning task comprised of watching a video on termites, suitable for the general public and an interview led question and answer session. This paper describes how FATHOM's comprehension classifier artificial neural network was trained and validated in comprehension detection using the standard backpropagation algorithm. The results show that high and low comprehension states can be detected from learner's non-verbal behavioural cues with testing classification accuracies above 76%.

Modelling e-Learner Comprehension Within a Conversational Intelligent Tutoring System

“…FATHOM [12] is a comprehension classification system which analyses learner nonverbal behaviour to automatically classify comprehension during tutorial questioning. The work demonstrates that video recordings of learners responding verbally to a human tutor can be computationally analysed to produce accurate classifications of learner comprehension levels during verbal student-tutor interactions.…”

“…Non-verbal behaviour (NVB) is any non-verbalised communicative behaviour including gestures, facial expressions, facial actions, physiological, chemical and audible information. Cutting edge research [12] suggests it is possible to model comprehension automatically by computer analysis of non-verbal behaviour. Buckingham et al [12] demonstrate that learner NVB can be used to model learner comprehension expressed during pre-recorded human to human interviews, conducted in a strictly controlled environment.…”

“…While the research [12,13] indicates that learner comprehension levels could be used as a feedback mechanism for adaptation within a CITS, doing so in a live realworld environment presents a number of novel challenges: a) Analysing live video in near real-time to track NVB in an uncontrolled environment (varying lighting, camera position), b) Accurately modelling comprehension from NVB during conversation with a virtual tutor, c) Producing accurate comprehension modelling without invasive bodyattached sensors or prohibitively expensive specialised equipment.…”

A hybrid model combining neural networks and decision tree for comprehension detection

“…To answer these research questions, two datasets collected from FATHOM studies have been used. The experimental study known as "Termites", reported in [15] was used to identify whether high and low human comprehension associated multi-channels of non-verbal behaviour reside within a video-recorded British (UK-based/English speaking) sample of participants in a class room environment. The Termites exploratory study builds upon lessons learned in prior work [3] where evidence was found that comprehension / noncomprehension could be detected in an African female population sample using a BPANN.…”

“…[15].Input to FATHOM is currently offline through recorded videos, which are streamed into FATHOM where a series of BPANN facial object locators, identify the location in a video frame of key visual features such as the eyes. For each nonverbal behavioural feature identified from a specific visual feature, the BPANN facial object pattern detectors identify its state i.e.…”