Multiple test batteries as predictors for pilot performance: A meta‐analytic investigation

Abstract

Test Batteries (TBs) have a long history of use in pilot selection. The extent to which TBs predict future pilot performance has important implications. The existing pilot‐related psychometric meta‐analyses have focused primarily on scores of individual ability tests, rather than the combined scores composited from multiple ability tests. The objective of this study was to investigate the predictive validity of TBs' composite scores for several criteria of pilot performance. Informed by the Cattell–Horn–Carroll theory, we proposed a classification scheme of six categories representing the most common composite scores in selection assessment: Acquired Knowledge, Perceptual Processing, Motor Abilities, Controlled Attention, General Ability, and Work Sample. For overall pilot performance, based on 267 correlations from 118 independent samples, results showed that the six categories of TBs are valid predictors (Meanr = .10–.34), and at least five of them have validity that is likely to generalize across selection contexts.

Predictive Validity of the Air Force Officer Qualifying Test (AFOQT) for Pilot Performance

“…Since our study focused primarily on the subtest level of the AFOQT, we did not attempt to correct the correlations for the common statistical artifacts (e.g., measurement error, range restriction), apart from the sampling error that is naturally accounted for by the metaanalysis procedure. This is consistent with prior psychometric meta-analyses of pilot selection tests (e.g., ALMamari & Traynor, 2019;Hunter & Burke, 1994;Martinussen, 1996), facilitating comparability of findings. All analyses were performed using the Excel-based version of the psychmeta R package (Dahlke & Wiernik, 2019).…”

“…In this case, it appears that a stand-alone test of a specific job-relevant ability can be as predictive as a composite score from multiple ability tests. The validity coefficient for the Pilot composite is slightly higher than the .15 and .16 found by Lynch (1991) for AFOQT and ASTB Pilot composites, respectively, and slightly lower than the .19 validity reported by ALMamari and Traynor (2019) for composite scores saturated with the broad factor of acquired knowledge. This observed validity of the Pilot composite, therefore, seems no exception from the general trend.…”

“…The study revealed a significant predictive role for three particular groups of predictors: a category containing a mix of cognitive abilities, tests category of psychomotor and information processing, and aviation information tests category. More broadly, the recent meta-analysis of ALMamari and Traynor (2019) introduced five categories for composite scores derived from multiple ability test batteries, each of which corresponded to five broad ability orientation, as well as work-sample form of testing (i.e., flight simulator). The syntheses of broad ability–flight performance relationships showed a more predictive role for the categories of work-sample, acquired knowledge, general ability, and motor abilities.…”

Predicting Commercial Pilot Training Performance

“…We expected that the paper-andpencil general mental ability (GMA) tests would predict pilot performance, especially GMA tests related to academic performance criteria, in accordance with previous findings (Martinussen, 1996). In addition, we expected that the computer-based tests assessing psychomotor coordination, spatial ability, and information processing would predict pilot performance, in line with previous meta-analyses (ALMamari & Traynor, 2019;Hunter & Burke, 1994;Martinussen, 1996) and earlier validation studies (Carretta et al, 2014;Lang-Ree et al, 2010;Martinussen & Torjussen, 2004). We expected small correlations between the Big Five traits and the performance criteria, in line with metaanalyses and single studies (Campbell, et al, 2009;Carretta et al, 2014).…”

A meta-analysis on air traffic controllers selection: cognitive and non-cognitive predictors

“…In a metaanalysis, Roth, Bobko, and McFarland (2005) estimated that work sample tests had a mean validity of .33. In addition, composite scores are used in selection psychology: multiple test scores combined and weighted into one score (Hattrup, 2012; for example, see ALMamari & Traynor, 2019). One limit of the composite score is its validity: it can be a type of black box or "rotten pot" (Bobko, Roth, & Buster, 2007).…”