Analysis of pilot feedback regarding the use of state awareness technologies during complex situations

Abstract

A flight simulation study was conducted at NASA Langley Research Center to evaluate flight deck systems that (1) predict aircraft energy state and/or autoflight configuration, (2) present the current state and expected future state of automated systems, and/or (3) show the state of flight-critical data systems in use by automated systems and primary flight instruments. Four new technology concepts were evaluated vis-à-vis current state-of-the-art flight deck systems and indicators. This human-in-the-loop study was conducted using commercial airline crews. Scenarios spanned a range of complex conditions and several emulated causal factors and complexity in recent accidents involving loss of state awareness by pilots (e.g. energy state, automation state, and/or system state). Data were collected via questionnaires administered after each flight, audio/video recordings, physiological data, head and eye tracking data, pilot control inputs, and researcher observations. This paper focuses specifically on findings derived from the questionnaire responses. It includes analysis of pilot subjective measures of complexity, decision making, workload, situation awareness, usability, and acceptability.

Flight simulation study of airplane state awareness and prediction technologies

“…The following sub-sections provide selected evidence supporting these hypotheses and claims. Additional results can be found in [9] [10]. In some areas, data analyses continue and will be published in future reports.…”

“…Below, these four technologies are briefly explained. Additional descriptions of their designs for this study, and findings resulting from their use can be found in [9] and [10].…”

“…Table III provides a summary of the number of technology usability assessments completed by the 22 pilots. Analysis of these assessments is provided in [10]. Participants were all Air Transport Pilot (ATP)-qualified and currently flying for major carriers (i.e., United, Delta, and American Airlines).…”

“…This consisted of the five-minute look-ahead "green line" on the VSD/ND, the circle symbols on this line representing where mode changes or energy-related problems are predicted to occur, and the predictive text messages indicated on the EICAS. Usability results are detailed in [10], but as shown in Figure 6, an average System Usability Scale (SUS) rating [20] of 80.35 and 78.95 was given to the TP technology for the pilot-flying (PF) and pilot-monitoring (PM) roles, respectively. This is not unexpected as the TRL is relatively high due to extensive previous research, testing, and subsequent refinement.…”

“…At the end of the two-day session, a semistructured interview allowed for more general observations and suggestions to help guide the research going forward. Some of this feedback is paraphrased below, with more specific data given in [10]. A more comprehensive listing will be published in the future.…”