Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Abstract

Active machine learning is a human-centric paradigm that leverages a small labelled dataset to build an initial weak classifier, that can then be improved over time through human-machine collaboration. As new unlabelled samples are observed, the machine can either provide a prediction, or query a human 'oracle' when the machine is not confident in its prediction. Of course, just as the machine may lack confidence, the same can also be true of a human 'oracle': humans are not all-knowing, untiring oracles. A human's ability to provide an accurate and confident response will often vary between queries, according to the duration of the current interaction, their level of engagement with the system, and the difficulty of the labelling task. This poses an important question of how uncertainty can be expressed and accounted for in a human-machine collaboration. In short, how can we facilitate a mutually-transparent collaboration between two uncertain actors-a person and a machine-that leads to an improved outcome? In this work, we demonstrate the benefit of human-machine collaboration within the process of active learning, where limited data samples are available or where labelling costs are high. To achieve this, we developed a visual analytics tool for active learning that promotes transparency, inspection, understanding and trust, of the learning process through human-machine collaboration. Fundamental to the notion of confidence, both parties can report their level of confidence during active learning tasks using the tool, such that this can be used to inform learning. Human confidence of labels can be accounted for by the machine, the machine can query for samples based on confidence measures, and the machine can report confidence of current predictions to the human, to further the trust and transparency between the collaborative parties. In particular, we find that this can improve the robustness of the classifier when incorrect sample labels are provided, due to unconfidence or fatigue. Reported confidences can also better inform human-machine sample selection in collaborative sampling. Our experimentation compares the impact of different selection strategies for acquiring samples: machine-driven, human-driven, and collaborative selection. We demonstrate how a collaborative approach can improve trust in the model robustness, achieving high accuracy and low user correction, with only limited data sample selections.

Collaborative behavior, performance and engagement with visual analytics tasks using mobile devices

“…Desolda et al [42] explored composition techniques to allow combining spatial-aware mobile devices to foster sensemaking activities. Phil et al [43] presented a visual analytic tool, ActiVAte, which can facilitate humanmachine collaboration, inspection, transparency, and trust of the learning process through human-machine collaboration. The VizCept system, for example, allows users to work in their own workspace and collaborate with others using representations like concept maps and timelines [44].…”

Explainable Embodied Agents Through Social Cues

“…We extrapolated the root terms and categorized them (e.g., explanation→explainability, expressive→expressivity). Assuming that papers with the same authors share the same definition, we grouped them by author, and [64,65] Predictability [32,67] Communicability [36,48] None [12,39,45,49,52] only report the most recent definition later in Table 6. When the definition was from an outside source not cited before, we mentioned both the definition and the reference.…”

“…Other studies in the literature do not refer to a specific capability of their system in the title but highlighted the application scenarios (e.g., autonomous driving [12], human robot teams in the army [80], interactive robot learning [52]) or mentioned behavioral dynamics [45] and humanmachine confidence [49].…”

“…This motivation empathizes the importance of communicating the agent's uncertainty, incapability, and existence of internally conflicting goals [44,62,64,76]. • Teamwork underlines the value of explainability in human-robot collaboration scenarios to build shared mental models and predict the embodied agent's behavior [16,34,36,49,65]. • Ethical decision making suggests that communicating the embodied agent's decisionmaking processes and capabilities, paired with situational awareness, increases a user's ability to make good decisions [1,44,61].…”

“…Providing information about the level of uncertainty and expressing robot incapability are core concepts of explainability that enhance human trust ("communicate uncertainty" [76], "provide appropriate measures of uncertainty" [62], "express robot incapability" [44]). The ability to anticipate an embodied agent's behavior and establish a two-way collaborative dialogue by identifying relevant differences between the humans' and the robots' model are shared elements of the definitions around teamwork ("anticipate robot's behavior" [36], "establish two-way collaborative dialogue" [49], "reconcile the relevant differences between the humans' and robot's model" [13], "share expectations" [34]). Authors who refer to ethical decision making identify the communication of intentions and context-dependent recommendations as crucial information ("robot's real capabilities, intentions, goals and limitations" [61], "context-dependent recommendations based on the level of trust and workload" [2]).…”