Pilot Study on the Reliability of the Coach's Eye: Identifying Talent Throughout a 4-Day Cadet Judo Camp

Abstract

A typical assumption found in talent identification literature is that different coaches, given the same athletes and circumstances, will identify the same subset of athletes as “talented”. However, while coaches play a major role during talent identification in practical sport settings, there is limited empirical research exploring the processes which underpin this. The purpose of this study was to explore the reliability of “the coach's eye” during the assessment of talent in a group of athletes. Specifically, this project compared inter-coach agreement between nine judo coaches (ages 35.8 ± 10.6 years) with varying levels of experience (12.9 ± 8.9 years) in the evaluation of 24 talented cadet judo athletes (13–15 years) at seven timepoints throughout a 4-day development training camp. Without discussion of their scores with other coaches, coaches provided a single score representing each athlete's “potential for future performance” on an 11-point Likert scale at each timepoint. Scores from each coach were converted into rankings from 1 to 24 to create a normalized scale to facilitate comparison of athletes. Based on their rankings at each timepoint, athletes were placed into one of three evenly distributed groups (high, medium, and low rank). Inter-coach agreement at each timepoint was determined by the number of coaches who ranked each athlete in the same group, categorized at three levels: 50, 75 or 100% agreement. Overall results showed that at completion of the camp, coaches reached 100% agreement on only two athletes, both of whom were in the high rank group. When inter-coach agreement was set at 50%, 15 athletes (62.5%) were placed into like groups. The first timepoint at which coaches were able to differentiate between the majority of athletes was Timepoint 3 (end of day 2). The findings suggest that, in isolation, coaches do not agree on the talent or potential of athletes. This indicates that the “coach's eye” is subjective and variable, and, given the same context, there is poor inter-coach agreement in the identification of talented athletes. In turn, these findings may have significant implications for both future talent identification research and athlete selection processes by sport organizations.

Coaches’ Assessment of Players Physical Performance: Subjective and Objective Measures are needed when Profiling Players

“…Secondly, the disparity in results may be due to coaches identifying and selecting players for a timepoint in the future versus current performance, and therefore, focussing on potential. Coaches are required to make judgements on players at a young age and select potential talent that may emerge in the future (Roberts et al, 2020). Indeed, coaches often referred to player's “potential” within their assessment, highlighting a focus toward the future.…”

“…However, it remains to be determined if rugby league and strength and conditioning (S&C) coaches can subjectively assess the physical performance of players, and the importance coaches place on certain attributes when assessing player performance and potential. Typically, subjective coach decisions are classified as the gold standard and have a significant role in talent identification (Roberts, Greenwood, Humberstone, & Raynor, 2020). With both rugby and S&C coaches being routinely involved in recruitment, decision making processes (Till, Muir, Abraham, Piggott, & Tee, 2019), talent identification, programming and long-term athletic development, such information is required to support multidimensional approaches within talent development.…”

Sources of information used by elite distance running coaches for selection decisions

“…This complexity is represented in the work done by Christensen [12], (and further supported empirically by Roberts et al, [16]), who hypothesised two different coaches will identify 'talent' differently. Recently, Jones and colleagues [17] also noted this finding, and concluded coaches conceptualize sporting 'talent' in different ways within different contexts.…”

Talent identification and development in judo: A perspective from Brazilian coaches

“…17 In judo, it has been found that the reliability of the coaches' eye to evaluate the potential of athletes is low when such conditions are not met. 25 Many studies have investigated the predictors of performance in judo, 8,9,12,26,27 including collecting the opinion of coaches. 13–15 However, few studies have considered biological maturation as one of the intervening variables in assessing the potential of young judo athletes.…”