Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Abstract

Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots (6 first officers, 6 captains, and 6 flight examiners). Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal (a) considerable variance within performance categories, (b) differences in the process used as evidence in support of a performance rating, (c) different numbers and types of facts (cues) identified, and (d) differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations.

A cross-sectional investigation of the relationships education, certification, and experience have with knowledge, skills, and abilities among aviation professionals

“…This important hiring pool allowed aviation hiring managers to hire from an experienced workforce. Experience is important in judging performance among pilots (Roth & Mavin, 2015). As such, experience is seen as one of the biggest obstacles new pilots face in gaining the appropriate experience that commercial operators desire (Ruiz, Voges & Mortag, 2006).…”

Flight Examiners’ Methods of Ascertaining Pilot Proficiency

“…This method was employed in a modified form that has been shown to be successful in recent aviation-related work (Roth & Mavin, 2015). Six pairs of flight examiners from two airlines were asked to assess the performances of both crewmembers (captain, first officer) appearing in three videotaped scenarios.…”

“…That is, even when flight examiners worked in pairsthereby decreasing individual subjectivity-varying assessment scores are observed (e.g., Weber, Mavin, Roth, & Dekker, in press). A previous study that mathematically modeled the scoring process accurately using fuzzy logic (e.g., Roth & Mavin, 2015) did not explain the reasons for such variations. The differences in flight examiners' overall documentary sense became intelligible in this study: Flight examiners differed in the contents of their observations (Tables 4 and 5), which entailed differences in the underlying constructs that were not given directly but indirectly through the manner in which they manifested themselves.…”

“…Early work on pilot performance assessment investigated assessment in terms of measurement models and focused on outcomes (e.g., Flin et al, 2003;Holt, Hansberger, & Boehm-Davis, 2002;O'Connor, Hörmann, Flin, Lodge, & Goeters, 2002), where the measurements sometimes were supported and enhanced by automated tools (e.g., Deaton et al, 2007;Johnston, Rushby, & Maclain, 2000). More recent studies focused on the nature of the evidence that flight examiners use in support of their ratings (e.g., Roth & Mavin, 2015). None of these studies provided data concerning the research question investigated here: How do flight examiners reason and which methods do they actually use and, importantly, make explicit to others as a rational basis for assessments of pilots' proficiency, knowledge, skills, or cognitive states?…”

“…A recent study from aviation provided evidence that assessors vary in the facts on which they base their assessment (Roth & Mavin, 2015). Moreover, first officers and regular captains go about assessment in ways that differed from flight examiners: The former move item by item through an assessment form and identify performances that allow them to give a score on that item, whereas the latter first construct narrative descriptions and then map the result onto the assessment form.…”