The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics

Abstract

Many evaluation methods have been used to assess the usefulness of Visual Analytics (VA) solutions. These methods stem from a variety of origins with different assumptions and goals, which cause confusion about their proofing capabilities. Moreover, the lack of discussion about the evaluation processes may limit our potential to develop new evaluation methods specialized for VA. In this paper, we present an analysis of evaluation methods that have been used to summatively evaluate VA solutions. We provide a survey and taxonomy of the evaluation methods that have appeared in the VAST literature in the past two years. We then analyze these methods in terms of validity and generalizability of their findings, as well as the feasibility of using them. We propose a new metric called summative quality to compare evaluation methods according to their ability to prove usefulness, and make recommendations for selecting evaluation methods based on their summative quality in the VA domain.

A Study on The Quality Evaluation of General Medical Education in China: An Empirical Analysis Based on 36 Medical Schools

“…Also known as summative evaluation, outcome evaluation is an assessment of the impact gained by the educated person, examining the cognitive changes in knowledge and skills, affect and behavior, and also the effectiveness of education in driving practice. [18] Taylor's behavioral goals model is one of the best known, and the concept of 'educational assessment' was first introduced and formally used by Taylor in his 'eightyear study', the Smith-Taylor Report. Taylor's behavioral goals model takes educational goals as its orientation, translates them into measurable behavioral goals for students, develops curricula, teaching materials or teaching programs based on these behavioral goals, carries out teaching activities, and then evaluates the effectiveness of teaching activities based on the behavioral goals, and judges the extent to which the actual teaching activities have achieved the desired goals through the evaluation of the final results.…”

QualDash: Adaptable Generation of Visualisation Dashboards for Healthcare Quality Improvement

“…Additional off-site activities aimed to predict the usability of the generated dashboards; and through on-site observations we sought to establish evidence of the usefulness of QualDash in a real-world healthcare QI setting. In this section, we describe the process and outcomes of 62 contact hours of evaluation.These activities were distributed across our project timeline, with each activity serving a summative evaluation goal, as defined by Khayata et al [23].…”

QualDash: Adaptable Generation of Visualisation Dashboards for Healthcare Quality Improvement

“…Additional off-site activities aimed to predict the usability of the generated dashboards; and through on-site observations we sought to establish evidence of the usefulness of QualDash in a real-world healthcare QI setting. In this section, we describe the process and outcomes of 62 contact hours of evaluation.These activities were distributed across our project timeline, with each activity serving a summative evaluation goal, as defined by Khayata et al [24].…”