Discovering the Sweet Spot of Human-Computer Configurations

Abstract

Interactive intelligent systems, i.e., interactive systems that employ AI technologies, are currently present in many parts of our social, public and political life. An issue reoccurring often in the development of these systems is the question regarding the level of appropriate human and computer contributions. Engineers and designers lack a way of systematically defining and delimiting possible options for designing such systems in terms of levels of automation. In this paper, we propose, apply and reflect on a method for human-computer configuration design. It supports the systematic investigation of the design space for developing an interactive intelligent system. We illustrate our method with a use case in the context of collaborative ideation. Here, we developed a tool for information extraction from idea content. A challenge was to find the right level of algorithmic support, whereby the quality of the information extraction should be as high as possible, but, at the same time, the human effort should be low. Such contradicting goals are often an issue in system development; thus, our method proposed helped us to conceptualize and explore the design space. Based on a critical reflection on our method application, we want to offer a complementary perspective to the value-centered design of interactive intelligent systems. Our overarching goal is to contribute to the design of so-called hybrid systems where humans and computers are partners. intelligent systems are AI 1 -supported systems, with which people interact when selecting songs, reading news or searching for products. Such systems should be examined by considering both the human and the system. Jameson and Riedl call this a 'binocular view' of interactive intelligent systems because the system's design includes algorithm design with interaction design, on the one hand, and a combined evaluation of a system's performance and human behavior, on the other hand [33]. However, existing research provides little guidance on how we should design interactive intelligent systems. Should a task be carried out by a human or a computer? What is an appropriate level of interaction vs. integration? Is human labor more preferable than automated action? How can we make an informed decision about allocating the task to either one or the other? How can we evaluate our decision? A proposal of a method for elaborating this spectrum is still missing.The authors have experienced this issue in the context of information extraction from ideas in the research area of collaborative ideation. In addition to complete manual approaches for making sense of an idea's content [4], algorithmic approaches were proposed which describe the content of the ideas statistically (e.g., [6]). However, both perspectives (manual vs. automatic) have their limitations; thus, the question is whether a 'sweet spot' that emphasizes the advantages of both perspectives exists and how such a 'sweet spot' can be identified. A 'sweet spot' defines a compromise between the often contradictory evaluative...

Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis

“…There's no transparency. P9's comment echoes P7's skepticism and highlights concerns related to control (e.g., "taking me out of the instrumentation"), something we see in prior work [25], as well as transparency [31,46,48]. AI that clusters data can act as an instrument for analysis.…”

Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface

“…Users are optimistic about the incorporation of AI-assisted decision making in data science [35] and pedagogical tools [1,31] and researchers are interested in the outcome of these interactions [21,35]. Joint human-AI decision making outcomes can be improved if user trust towards the AI is calibrated appropriately [42] and when there is an appropriate level of contribution from each party in a human-AI collaborative context [18,19]. An implicit part of AI-assisted decision making is how the AI system's results are presented to the user.…”

Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis

“…Human-AI collaboration itself is increasingly gaining importance in CSCW scholarship. Emerging work focuses on a variety of topics from information extraction [28] to image understanding [44], consistently highlighting the need for transparency and explainability of AI. Mackeprang et al specifically pointed out that at high levels of automation, users were unlikely to challenge results provided by AI when unaccompanied by explanations [28].…”

“…Emerging work focuses on a variety of topics from information extraction [28] to image understanding [44], consistently highlighting the need for transparency and explainability of AI. Mackeprang et al specifically pointed out that at high levels of automation, users were unlikely to challenge results provided by AI when unaccompanied by explanations [28]. This situation, to say the least, would be undesirable in the context of qualitative research.…”