Toward harnessing user feedback for machine learning

Abstract

There has been little research into how end users might be able to communicate advice to machine learning systems. If this resource-the users themselves-could somehow work hand-in-hand with machine learning systems, the accuracy of learning systems could be improved and the users' understanding and trust of the system could improve as well. We conducted a think-aloud study to see how willing users were to provide feedback and to understand what kinds of feedback users could give. Users were shown explanations of machine learning predictions and asked to provide feedback to improve the predictions. We found that users had no difficulty providing generous amounts of feedback. The kinds of feedback ranged from suggestions for reweighting of features to proposals for new features, feature combinations, relational features, and wholesale changes to the learning algorithm. The results show that user feedback has the potential to significantly improve machine learning systems, but that learning algorithms need to be extended in several ways to be able to assimilate this feedback.

Paper or Pixel? Comparing Paper- and Tool-Based Participatory Design Approaches

“…Various coding schemes have been developed for a variety of topics, ranging from user comments on machine learning [19], over student comments on the teaching performance of professors [20] to YouTube comments [21]. [22] applied PD in a school setting to plan and improve lessons together with students and developed a coding scheme to code the spoken comments made during discussions.…”

Using Decision-Theoretic Experience Sampling to Build Personalized Mobile Phone Interruption Models

“…Many different experience sampling techniques have been proposed to accurately elicit data labels from users in order to build classifiers including diary studies [3], device-initiated questions at different intervals of time [10,20], and based on contextawareness [11] and previous labels [26]. The active learning literature have also proposed a variety of ways to choose which data should be labeled [1,12,17,18].…”

End-user interactions with intelligent and autonomous systems

“…Recent attempts have moved beyond explaining rule-based systems [25] toward more complex algorithms [20,26]. Examples of explanations for specific decisions include why… and why not… descriptions [14,16], visual depictions of the assistant's known correct predictions versus its known failures [29], confidence of the system in making predictions [13,19], and electronic "door tags" displaying predictions of worker interruptibility with the reasons (e.g., "talking detected" [31]).…”