Relationships between Contextual and Task Performance and Interrater Agreement: Are There Any?

Díaz-Vilela, Luis; Delgado, Naira; Isla-Díaz, Rosa; Díaz-Cabrera, Dolores; Hernández-Fernaud, Estefanía; Sánchez, Christian Rosales

doi:10.1371/journal.pone.0139898

Cited by 30 publications

(25 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Also, they did not find evidence that job performance can be expected to be highly skewed. Also, Díaz-Vilela et al (2015) found a normal distribution of administrative ratings of task and contextual performance in a sample of Spanish public servants. Based on these findings, it can be reasonably assumed that supervisory performance ratings are normally distributed and, therefore, that the reliability correction formula and the selection ratio formula can be applied in the present study.…”

Section: Methodsmentioning

confidence: 94%

Meta-Analysis of Interrater Reliability of Supervisory Performance Ratings: Effects of Appraisal Purpose, Scale Type, and Range Restriction

Salgado

Moscoso

2019

Front. Psychol.

View full text Add to dashboard Cite

Objectives: This reliability generalization study aimed to estimate the mean and variance of the interrater reliability coefficients (ryy) of supervisory ratings of overall, task, contextual, and positive job performance. The moderating effect of the appraisal purpose and the scale type was examined. It was hypothesized that the ratings collected for research purposes and multi-item scales have higher ryy. It was also examined whether ryy was similar for the four performance dimensions.Method: A database consisting of 224 independent samples was created and hierarchical sub-grouping meta-analyses were conducted.Results: The appraisal purpose was a moderator of ryy for the four performance dimensions. Scale type was a moderator of ryy for overall and task performance collected for research purposes. The findings also suggest that supervisors seem to have less difficulty evaluating overall job performance than task, contextual, and positive performance. The best estimates of the observed ryy for overall job performance are 0.61 for research-collected ratings and 0.45 for administrative-collected ratings.Conclusions: (1) Appraisal purpose moderates ryy and researchers and practitioners should be aware of its effects before collecting ratings or using empirically-derived interrater reliability distributions, (2) Scale type seems to moderate ryy in the case of the ratings collected for research purposes, only, (3) overall job performance is more reliably rated than task, contextual, and positive performance. Implications for research and practice are discussed.

show abstract

Section: Methodsmentioning

confidence: 94%

Meta-Analysis of Interrater Reliability of Supervisory Performance Ratings: Effects of Appraisal Purpose, Scale Type, and Range Restriction

Salgado

Moscoso

2019

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Task performance assessment scale : this tool was composed of 14 items based on the task inventory of a previous job analysis [44]. The participants used a computer application to evaluate the quality and frequency with which the main characters in each video performed the tasks associated with their jobs [45].…”

Section: Methodsmentioning

confidence: 99%

Does effectiveness in performance appraisal improve with rater training?

2019

Self Cite

View full text Add to dashboard Cite

Performance appraisal is a complex process by which an organization can determine the extent to which employees are performing their work effectively. However, this appraisal may not be accurate if there is no reduction in the impact of problems caused by possibly subjective rater judgements. The main objective of this work is to check the effectiveness—separately and jointly—of the following four training programmes in the extant literature aimed at improving the accuracy of performance assessment: 1) Performance Dimension Training, 2) Frame-of-Reference, 3) Rater Error Training, and 4) Behavioural Observation Training. Based on these training strategies, three programmes were designed and applied separately. A fourth programme was a combination of the other three. We analyzed two studies using different samples (85 students and 42 employees) for the existence of differences in the levels of knowledge of performance and its dimensions, rater errors, observational accuracy, and accuracy of task and citizenship performance appraisal, according to the type of training raters receive. First, the main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes (Training_4_programmes), increases the level of knowledge of performance and its dimensions. Second, groups that receive training in rater error score higher in knowledge of biases than the other groups, whether or not they have received training. Third, participants’ observational accuracy improves with each new moment measure (post-training and follow-up), though not because of the type of training received. Fourth, participants who receive training through the programme that combine the other four gave a task performance appraisal that was closer to the one undertaken by the judges-experts than the other groups. And finally, students’ citizenship performance appraisal does not vary according to type of training or to different moment measures, whereas the group of employees who received all four types of training gave a more accurate citizenship performance assessment.

show abstract

“…These performance samples included a series of performance task activities habitually carried out by administrative and office staff, as well as the contextual behaviours of the employee in work interactions. These administrative tasks were identified by analyzing a previous post (Díaz-Vilela et al, 2015). In relation to contextual behaviours, according to an assessment by a group of experts and administrative professionals, the ten most representative and realistic descriptions were selected from the 20 created, in line with Borman and Motowidlo's (1993) that such courses were useless.…”

Section: Materials and Toolsmentioning

confidence: 99%

Influence of measurement type and the moment of occurrence of low performance behaviour's on task and citizenship performance appraisal

Rosales¹,

Díaz-Cabrera²,

Hernández-Fernaud³

2021

RIP/IJP

View full text Add to dashboard Cite

This research studies whether the moment of occurrence of a task or contextual behaviour with a low performance produces a primacy or recency effect and whether it causes changes in performance appraisal. We also analyzed whether the nature of assessment questionnaire items affects raters’ assessments and how the sequence of questionnaire presentation and completion may do so. Participants were 146 undergraduate students. We used a design with two inter-subject variables (questionnaire presentation and performance sequence) and one within-subject variable (global versus specific questionnaires). Findings show that if a low performance is presented at the beginning of the assessment period, the performance assessment will be more negative. Also, results indicate that task performance appraisals and contextual behaviour assessments are higher and less accurate when performed with a questionnaire that includes global items.

show abstract

Relationships between Contextual and Task Performance and Interrater Agreement: Are There Any?

Cited by 30 publications

References 34 publications

Meta-Analysis of Interrater Reliability of Supervisory Performance Ratings: Effects of Appraisal Purpose, Scale Type, and Range Restriction

Meta-Analysis of Interrater Reliability of Supervisory Performance Ratings: Effects of Appraisal Purpose, Scale Type, and Range Restriction

Does effectiveness in performance appraisal improve with rater training?

Influence of measurement type and the moment of occurrence of low performance behaviour's on task and citizenship performance appraisal

Contact Info

Product

Resources

About