Controller Fatigue State Detection Based on ES-DFNN

Abstract

The fatiguing work of air traffic controllers inevitably threatens air traffic safety. Determining whether eyes are in an open or closed state is currently the main method for detecting fatigue in air traffic controllers. Here, an eye state recognition model based on deep-fusion neural networks is proposed for determination of the fatigue state of controllers. This method uses transfer learning strategies to pre-train deep neural networks and deep convolutional neural networks and performs network fusion at the decision-making layer. The fused network demonstrated an improved ability to classify the target domain dataset. First, a deep-cascaded neural network algorithm was used to realize face detection and eye positioning. Second, according to the eye selection mechanism, the pictures of the eyes to be tested were cropped and passed into the deep-fusion neural network to determine the eye state. Finally, the PERCLOS indicator was combined to detect the fatigue state of the controller. On the ZJU, CEW and ATCE datasets, the accuracy, F1 score and AUC values of different networks were compared, and, on the ZJU and CEW datasets, the recognition accuracy and AUC values among different methods were evaluated based on a comparative experiment. The experimental results show that the deep-fusion neural network model demonstrated better performance than the other assessed network models. When applied to the controller eye dataset, the recognition accuracy was 98.44%, and the recognition accuracy for the test video was 97.30%.

An Eye State Recognition System Using Transfer Learning: AlexNet-Based Deep Convolutional Neural Network

“…The studies with the closest accuracy value to the proposed DCNN architecture were the studies of Liu et al [26], Zhao et al [22] and Liang et al [29], as seen in Table 8. CNN based on deep learning, as in the proposed DCNN architecture, was one of the strategies used in these investigations.…”

“…10, DCNN achieved the best performance in the accuracy metric. In the AUC value of the proposed method, it took the second place after Liang et al [29] with a slight difference. The DFNN method proposed by Liang et al evaluated in both CEW and ZJU datasets as given in Tables 8 and 9.…”

“…This method was validated on three eye condition datasets (ZJU, CEW, and MRL) and experimental results indicated that the proposed DCNNE method showed remarkable success in CEW and ZJU dataset compared to the state-of-the-art methods in the literature. Liang et al [29] used the deep-fusion neural network (DFNN) model, which is formed by combining the deep neural network, which extracts the vector features of the eye, and the deep convolutional neural network, which extracts the tissue features, to increase the detection efficiency and accuracy in the detection of eye fatigue in controllers. The proposed method was evaluated in ZJU, CEW and ATCE eye state datasets, and the comparative results showed that DFNN outperformed the early technologies used in eye state recognition.…”

“…Because of the superior performance of CNN in image classification, one of the sub-branches of machine learning has displayed a considerable effect in many imagebased applications [17]. As a consequence, in research by Liu et al [26], Huang et al [10], Saurav et al [28] and Liang et al [29], deep learning-based CNN approaches, have gotten started to be preferred in eye state recognition. Unlike other machine learning approaches, CNN's multi-layered structure does not demand meticulous engineering or knowledge because it yields representational learning from its raw data.…”

Analyzing Multi-Mode Fatigue Information from Speech and Gaze Data from Air Traffic Controllers

“…Recently, speech-signal research has led to scholars concluding that the process of generating speech signals is a nonlinear process; that is, it is neither a deterministic linear sequence nor a random sequence, but rather a nonlinear sequence with chaotic components. Therefore, traditional linear filter models cannot fully represent the information contained in speech signals [11]. A nonlinear dynamic model of a speech signal is generally constructed using a delay-phase diagram that is obtained by reconstructing the time series of a one-dimensional speech signal in the phase space [12].…”