A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents

Abstract

Sleep-deprived fatigued person is likely to commit more errors that may even prove to be fatal. Thus, it is necessary to recognize this fatigue. The novelty of the proposed research work for the detection of this fatigue is that it is nonintrusive and based on multimodal feature fusion. In the proposed methodology, fatigue is detected by obtaining features from four domains: visual images, thermal images, keystroke dynamics, and voice features. In the proposed methodology, the samples of a volunteer (subject) are obtained from all four domains for feature extraction, and empirical weights are assigned to the four different domains. Young, healthy volunteers (n = 60) between the age group of 20 to 30 years participated in the experimental study. Further, they abstained from the consumption of alcohol, caffeine, or other drugs impacting their sleep pattern during the study. Through this multimodal technique, appropriate weights are given to the features obtained from the four domains. The results are compared with k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers. The proposed nonintrusive technique has obtained an average detection accuracy of 93.33% in 3-fold cross-validation.

Series Arc Fault Detection Based on Multimodal Feature Fusion

“…Wavelet packet transform is an extension of wavelet transform. Orthogonal wavelet packet bases are generated from a standard Orthogonalization scale parameter through a two-scale difference equation [ 21 , 22 ]. The calculation of the orthogonal wavelet packet bases is shown in Equation (3), and the biorthogonal difference equation is represented by Equation (4).…”

“…Wang [ 21 ] reduces the sampling error and improves the efficiency and accuracy of a vehicle detection system by matching the coordinate relationship between the two sensors. Virk [ 22 ] uses this multimodal technique to assign appropriate weights to the features obtained in multiple domains, thus improving the detection accuracy. Liang [ 23 ] improves the accuracy of false information detection by integrating text features and visual features and taking low-dimensional features as joint features of multiple modes.…”