Multimodality Inferring of Human Cognitive States Based on Integration of Neuro-Fuzzy Network and Information Fusion Techniques

Abstract

To achieve an effective and safe operation on the machine system where the human interacts with the machine mutually, there is a need for the machine to understand the human state, especially cognitive state, when the human's operation task demands an intensive cognitive activity. Due to a well-known fact with the human being, a highly uncertain cognitive state and behavior as well as expressions or cues, the recent trend to infer the human state is to consider multimodality features of the human operator. In this paper, we present a method for multimodality inferring of human cognitive states by integrating neuro-fuzzy network and information fusion techniques. To demonstrate the effectiveness of this method, we take the driver fatigue detection as an example. The proposed method has, in particular, the following new features. First, human expressions are classified into four categories: (i) casual or contextual feature, (ii) contact feature, (iii) contactless feature, and (iv) performance feature. Second, the fuzzy neural network technique, in particular Takagi-Sugeno-Kang (TSK) model, is employed to cope with uncertain behaviors. Third, the sensor fusion technique, in particular ordered weighted aggregation (OWA), is integrated with the TSK model in such a way that cues are taken as inputs to the TSK model, and then the outputs of the TSK are fused by the OWA which gives outputs corresponding to particular cognitive states under interest (e.g., fatigue). We call this method TSK-OWA. Validation of the TSK-OWA, performed in the Northeastern University vehicle drive simulator, has shown that the proposed method is promising to be a general tool for human cognitive state inferring and a special tool for the driver fatigue detection.

A driver fatigue recognition model based on information fusion and dynamic Bayesian network

“…Therefore, the LF/HF ratio describing driver's states was taken as one of the observable variables corresponding to the nodes of the DBN graph. In this paper, the method presented by Yang et al [9] is adapted to calculate the changes around the standard baseline of EEG.…”

“…The frequency domain of EEG mainly includes the delta band (0.5-4 Hz) corresponding to the sleep activity, the theta band (4-7 Hz) related with drowsiness, the alpha band (8)(9)(10)(11)(12)(13) Hz) corresponding to relaxation and creativity, and the beta band (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25) corresponding to activity and alertness [17]. Note that only the alpha band has strong relations with fatigue study, and the variations in the EEG trace, such as a decrease in the alpha rhythms, is interpreted to indicate states of fatigue [20,31].…”

“…In the present study, a simple low-pass filter with a cuto? frequency of 50 Hz and the independent component analysis are employed to smoothen the EEG signals, the smoothened signals are then transformed into the frequency domain by use of the fast Fourier transform (FFT) algorithm (for further details on calculating the change around the standard baseline of EEG, see Yang et al [9]). …”

Engineering Issues in Physiological Computing

“…Context-awareness does seem to be gaining popularity, especially in fatigue studies. Ji et al (2006) and Yang et al (2008) both combined physiological measurements of fatigue with user conditions (e.g. sleep quality, workload) as well as environmental conditions (e.g.…”

“…Kalman filters, which are commonly used in general sensor fusion and have been shown to improve psychophysiological inference with autonomic nervous system responses by 'learning' about a subject over time (Koenig et al 2011;Novak et al 2011), are theoretically a simple dynamic Bayesian network, though not wellvalidated in physiological computing. More advanced dynamic Bayesian networks have been tested, some of them incorporating context-awareness (Ji et al 2006;Lee and Chung 2012;Yang et al 2008). Besides Bayesian networks, alternate dynamic classifiers include e.g.…”