A fuzzy set theory based computational model to represent the quality of inter-rater agreement

Assessment of Nontechnical Skills

“…The reliable and valid assessment of pilot performance, including crew resource management (non-technical) skills, despite good efforts to address variation through training, continues to be a difficult area in the aviation industry in the face of efforts by the International Civil Aviation Organization to make it an integral aspect of global quality assurance. Traditional approaches to inter-rater reliability are associated with considerable problems (e.g., Ciavolino et al, 2013;Hallgren, 2012). Much of the effort conducted so far has been concerned developing methods that decrease variation attributed to rater error.…”

“…Categorization that uses approximate descriptions (e.g., poor, satisfactory, and very good) can be meticulously modeled-as shown in mathematical approaches to medical diagnosis (e.g., Klir & Yuan, 1995;Innocent, John & Garibaldi, 2004) and medical expert systems (Adlassnig, 1986;Phuong, 1995)-using fuzzy logic. We thereby follow others, though using a differing approach, in choosing fuzzy logic to address the weaknesses of the traditional approach to inter-rater reliability (e.g., Ciavolino, Salvatore, & Calgagnì, 2013) and issues in personnel assessment (e.g., Capaldo & Zollo, 2001). We exemplify the approach with a case from our research in which pilots give their reasons for assessing the performances of peers videotaped in flight simulator scenarios, using as our example those parts of the assessment sessions that focused on situational awareness.…”

Good reasons for high variability (low inter-rater reliability) in performance assessment: Toward a fuzzy logic model

“…In the literature, there have been suggestions that (a) performance assessment in the field is a categorization and judgment rather than measurement issue (Govaerts et al, 2007); and (b) categorization and judgments generally-as in medical diagnosis (Esogbue & Elder, 1980)-and performance assessment specifically is well modeled by fuzzy logic reasoning (e.g. Ciavolino et al, 2013;Özdaban & Özkan, 2010;Roth & Mavin, 2013). In the following, we articulate how the results presented in Table 2 are modeled using a fuzzy logic approach.…”

“…In the fuzzy logic model proposed for performance assessment (Ciavolino et al, 2013;Özdaban & Özkan, 2010;Roth & Mavin, 2013), two fuzzy sets BL and BU (0 ≤ Bi,j ≤ 1) define the lower and upper boundary of the five performance levels associated with each category used in the assessment. Because performance raters have been reported to tend to avoid extremes and exhibit a preference for the middle range, we defined the boundaries for unsatisfactory (u), minimal (m), satisfactory (s), good (g), and very good (vg) as {0 ≤ u < .15 ≤ m < .35 ≤ s < .65 ≤ g .85 ≤ vg ≤ 1} (i.e., a satisfactory rating has BL,s = .35 and BU,s = .65 as lower and upper boundary).…”

A Generalized Maximum Entropy (GME) estimation approach to fuzzy regression model

“…These can be described according to the nature of the input data, the method and the output data considered (input-method-output schema). In particular, in the first approach, named fuzzy-crisp-crisp, the fuzzy input dataX are transformed into crisp data (e.g., by means of some defuzzification procedures) and standard statistical methods are used to perform data analysis (e.g., crisp least squares) [11]. The resulting output of this procedure are also crisp data y.…”