Towards automated assessment of team performance by mimicking expert observers’ ratings

Abstract

Automation is the holy grail of performance assessment. Cheap and reliable automated systems that produce consistent feedback on performance. Many such systems have been proposed that accurately measure the state of a product or the outcome of a process. Procedural faults can be detected and even mitigated without the need for human interference. In production industry and professional sports, this is a natural part of business. However, in macrocognitive team performance studies, human appraisal is still king. This study investigates the reliability of human observers as assessors of performance among virtual teams, and what they base their assessments on when only able to monitor one of the team members at a time. The results show that expert observers put a lot of emphasis on task outcomes and on communication and are generally reliable raters of team performance, but there are several aspects that they cannot rate reliably under these circumstances, e.g., team workload, stress, and collaborative problem-solving. Through simple algorithms, this study shows that by capturing task scores and different quantitative communication metrics, team performance ratings can be estimated to closely match how the expert observers assess team performance in a virtual team setting. The implication of the study is that numeric team performance estimations can be acquired by automated systems, with reasonable accuracy and reliability compared to observer ratings.

Assessing the Utility of a Quality-of-Care Assessment Tool Used in Assessing Comprehensive Care Services Provided by Community Health Workers in South Africa

“…The communication items also, may be more subjective than the messages scores that are in line with the South African CHW training manual ( 39 ). This is because communication could be affected by stress, emotion and workload and may require additional training of the observers on speech recognition ( 61 ). These results suggest that the quality-of-care messages score may be useful summative assessment, but this would provide a partial assessment.…”

Personal Competition Among Sports Players and Their Performance as a Team: A Moderated Mediation Model

“…The action regulation theory describes the ways people govern their objective conduct via many activities or procedures, including goal formation, externally and internally orientation, preparation, implementation, and assessment. Work planning, cooperation in tasks (i.e., collaboration and knowledge exchange), work evaluation (procedures and quality assurance) ( Frese and Zapf, 1994 ; Tschan, 2002 ; Rousseau et al, 2006 ), and adaptation behaviors are all behaviors that assure team performance (such as backing-up, coaching, problem-solving, and team innovative behavior) ( Rousseau et al, 2006 ; Granåsen, 2019 ).…”