There is no perfect evaluator: An investigation based on prospect theory

Abstract

Psychophysical measurement has been widely used in various scientific and engineering fields.In the field of human-computer interaction (HCI), many prominent constructs, such as usability, affective satisfaction, user value, and user experience, have been assessed by psychophysical measurement, including Likert, Borg, and 0-100 scales. In spite of considerable empirical evidence that psychophysical measurement is reliable and valid, these measures also have fundamental limitations, such as distortion at both extremes. This study introduced prospect theory, which compensates for decision-making behavior in uncertain situations, and applied the theory to the transformation of assessed data. In a typical usability assessment study with target selection tasks, this study revealed that psychophysically assessed data might improve reliability and validity. In addition, this study showed that people tend to regard around 75 as judgment criteria on a 0-100 scale. The results of this study can be used and applied to improve the reliability and validity of subjective measurements. This study suggested a method to improve psychological measurements based on prospect theory. The method is expected to be applied to practical engineering experiments, which has been widely conducted in the field of human factors and HCI. K E Y W O R D Scognitive models, human error analysis, psychometrics, qualitative methods, scaling, usability testing and evaluation

Compensating Qualitative Rating Distortion of User Experience Evaluation Based on Prospect Theory

“…Various data analysis can confirm this indirectly. This study accepts the results of previous studies [8] and assumes that humans are based on specific scores when evaluating something. As a result, 75 points on the 0-100 scale, which were important criteria in previous studies, appear to have worked in this study.…”

“…This study tried to confirm a methodology to make compensations in psychophysical measurement using prospect theory in two case studies. First, a prospect theory-based approach was originally verified in a previous study through a usability assessment experiment consisting of serial target selection tasks [8]. In order to enlarge the base and ensure the methodology per se, two case studies were used in this study, targeting affect and user value concepts.…”

“…As mentioned in the first case study, scores for the questionnaire items were transformed using a prospect theory value function, which is based on the method described in Section 3.2.2 and the previous study [8]. Five score sets based on five reference scores (i.e., 0, 25, 50, 75, and 100) were investigated using confirmatory factor analysis.…”

“…At the extremes, the scale is bound to be distorted. Second, when linguistic anchors exist, for example, even if 50 points are given at a neutral level on the 0-100 scale, people tend to be based on slightly higher scores [8]. Although it may not be a problem to allow some level distortion and analyze the data in this way, it goes without saying that it may be better to analyze the data after correction.…”

“…A previous study, designed to be preliminary to this study, intended to compensate for limitations in psychophysical measurement by using prospect theory-based transformation [8]. The theory was based on human behavior in uncertain situations including diminishing sensitivity and loss aversion.…”