A Cyber-Physical-Human System for One-to-Many UAS Operations: Cognitive Load Analysis

Planke, Lars J.; Lim, Yixiang; Gardi, Alessandro; Sabatini, Roberto; Kistan, Trevor; Ezer, Neta

doi:10.3390/s20195467

Cited by 12 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Subsequently, a de-briefing was carried out where participants provided feedback on the mission along with subjective ratings for their workload and situational awareness in each of the three phases using a 10-point Likert scale. The workload and situational awareness were found to be significant across the three phases [33].…”

Section: Methodsmentioning

confidence: 87%

“…The concept of operations (CONOPS), experimental scenario, procedure, and participants have been previously presented and discussed [33,36,37]. In brief, participants assume the role of a remote pilot tasked with coordinating multiple UAVs in a bushfire surveillance mission.…”

Section: Multi-uav Scenariomentioning

confidence: 99%

“…To define the adaptation logic, specific mission tasks are first defined along with the relevant cognitive states that are affected when performing such tasks (e.g., workload, attention, situational awareness), as well as the neurophysiological observables required to infer the user's cognitive states. As individuals have different neurophysiological responses to external stimuli, an offline calibration phase is required to train the cognitive models used in the inference module before they are deployed for real-time prediction and adaptation [33]. In this phase, user-specific neurophysiological and task performance data from previous runs (with each run typically lasting a half-hour to an hour) are used to calibrate the cognitive model.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

et al. 2021

Self Cite

View full text Add to dashboard Cite

Advances in unmanned aircraft systems (UAS) have paved the way for progressively higher levels of intelligence and autonomy, supporting new modes of operation, such as the one-to-many (OTM) concept, where a single human operator is responsible for monitoring and coordinating the tasks of multiple unmanned aerial vehicles (UAVs). This paper presents the development and evaluation of cognitive human-machine interfaces and interactions (CHMI2) supporting adaptive automation in OTM applications. A CHMI2 system comprises a network of neurophysiological sensors and machine-learning based models for inferring user cognitive states, as well as the adaptation engine containing a set of transition logics for control/display functions and discrete autonomy levels. Models of the user’s cognitive states are trained on past performance and neurophysiological data during an offline calibration phase, and subsequently used in the online adaptation phase for real-time inference of these cognitive states. To investigate adaptive automation in OTM applications, a scenario involving bushfire detection was developed where a single human operator is responsible for tasking multiple UAV platforms to search for and localize bushfires over a wide area. We present the architecture and design of the UAS simulation environment that was developed, together with various human-machine interface (HMI) formats and functions, to evaluate the CHMI2 system’s feasibility through human-in-the-loop (HITL) experiments. The CHMI2 module was subsequently integrated into the simulation environment, providing the sensing, inference, and adaptation capabilities needed to realise adaptive automation. HITL experiments were performed to verify the CHMI2 module’s functionalities in the offline calibration and online adaptation phases. In particular, results from the online adaptation phase showed that the system was able to support real-time inference and human-machine interface and interaction (HMI2) adaptation. However, the accuracy of the inferred workload was variable across the different participants (with a root mean squared error (RMSE) ranging from 0.2 to 0.6), partly due to the reduced number of neurophysiological features available as real-time inputs and also due to limited training stages in the offline calibration phase. To improve the performance of the system, future work will investigate the use of alternative machine learning techniques, additional neurophysiological input features, and a more extensive training stage.

show abstract

Section: Methodsmentioning

confidence: 87%

Section: Multi-uav Scenariomentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Planke та співавт. [8] у своїх дослідженнях показують наявність кореляції професійного навантаження з досліджуваними фізіологічними показниками, зокрема негативні наслідки виникають із дедалі складнішою автоматизацією системи, наприклад, втрата ситуативної обізнаності операторів і збільшення або зменшення розумового навантаження. Значною мірою на ефективність професійної діяльності операторів впливають фактори, що сприяють розвитку психоемоційної напруги у працюючого, і стресостійкість є важливою особистісною якістю [9,10].…”

Section: вступunclassified

Features of Application of Psychophysiological Profiles of Operators of Unmanned Aviation Complexes for Their Occupational Selection

Kalnysh,

Shvets,

Maltsev

et al. 2023

Fiziol Zh

View full text Add to dashboard Cite

In order to develop new approaches to the psychophysiological selection of operators, 219 male servicemen aged 20 to 40 years old with experience in operating unmanned aerial systems of the first class Light and involved in a wide range of professional tasks were examined. To study their psychophysiological status, modified methods were used using the PFI-2 hardware and software complex. The following indicators were determined: accuracy of reaction to a moving object, strength of nervous processes, functional mobility of nervous processes, simple visual-motor reaction, complex visual-motor reaction, orientation in space, and visual memory. To identify a set of informative indicators that can best divide the analyzed group of operators into 3 clusters, a stepwise discriminant analysis was applied. The technique was developed to obtain several psychophysiological profiles of vocational aptitude, equally effective in terms of content. The statistical discrepancy of these profiles has been established. Ways of evaluating the superiority of these profiles have been shown. The principle of “necessary diversity” in the implementation of occupational psychophysiological selection of operators working in conditions with increased danger has been proposed and discussed. A number of advantages of using this principle to increase accuracy and expand the contingent of persons subject to occupational psychophysiological selection have been identified. The technology of graphical presentation of psychophysiological profiles of operators of unmanned aerial complexes was developed for the purpose of assessment of suitability for professional tasks.

show abstract

“…The integration of the CHMS can support aerospace systems to operate at higher levels of automation while ensuring that the human operator maintains a central role within the system and that trust in the system is maintained. Moreover, the CHMS can play an important role in the emergence of new operational aerospace systems, such as the transition from two pilots to Single Pilot Operation (SPO) in commercial aircrafts [7,61], One-To-Many (OTM) Unmanned Aerial Vehicle (UAV) operation [52,62], evolution of ATM [63] and Urban Traffic Management (UTM) [64].…”

Section: Chms Frameworkmentioning

confidence: 99%

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

show abstract

A Cyber-Physical-Human System for One-to-Many UAS Operations: Cognitive Load Analysis

Cited by 12 publications

References 40 publications

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

Features of Application of Psychophysiological Profiles of Operators of Unmanned Aviation Complexes for Their Occupational Selection

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

Contact Info

Product

Resources

About