Assessing Open-Ended Human-Computer Collaboration Systems: Applying a Hallmarks Approach

Abstract

There is a growing desire to create computer systems that can collaborate with humans on complex, open-ended activities. These activities typically have no set completion criteria and frequently involve multimodal communication, extensive world knowledge, creativity, and building structures or compositions through multiple steps. Because these systems differ from question and answer (Q&A) systems, chatbots, and simple task-oriented assistants, new methods for evaluating such collaborative computer systems are needed. Here, we present a set of criteria for evaluating these systems, called Hallmarks of Human-Machine Collaboration. The Hallmarks build on the success of heuristic evaluation used by the user interface community and past evaluation techniques used in the spoken language and chatbot communities. They consist of observable characteristics indicative of successful collaborative communication, grouped into eight high-level properties: robustness; habitability; mutual contribution of meaningful content; context-awareness; consistent human engagement; provision of rationale; use of elementary concepts to teach and learn new concepts; and successful collaboration. We present examples of how we used these Hallmarks in the DARPA Communicating with Computers (CwC) program to evaluate diverse activities, including story and music generation, interactive building with blocks, and exploration of molecular mechanisms in cancer. We used the Hallmarks as guides for developers and as diagnostics, assessing systems with the Hallmarks to identify strengths and opportunities for improvement using logs from user studies, surveying the human partner, third-party review of creative products, and direct tests. Informal feedback from CwC technology developers indicates that the use of the Hallmarks for program evaluation helped guide development. The Hallmarks also made it possible to identify areas of progress and major gaps in developing systems where the machine is an equal, creative partner.

Importance of the Chatbot in the Different Social Areas

“…En el campo médico, los chatbots se están utilizando para ayudar a los pacientes a obtener información sobre su salud y tratamiento, así como para brindar seguimiento y recordatorios de citas médicas y medicamentos recetados (21).…”

Overview of the COVID-19 text mining tool interactive demonstration track in BioCreative VII

“…This approach assesses the experiences and reactions of users exercising an ‘individual’ system for typical tasks. This approach has been used ( 31 ) to drive community progress toward systems that collaborate with humans on a variety of complex, open-ended activities. The usability-based approach made it possible to provide system-specific feedback to developers—a major goal of the BioCreative demo task.…”

Affordance embeddings for situated language understanding

“…Namely, it is one where each interlocutor has something "interesting" to say, and one that enables them to work together to achieve common goals and build off each other's contributions, thereby conveying the impression to the user that the computer system is experiencing the same events. Hence, the foundation of multimodal communication, be it human-human or humancomputer, is based on the following criteria (Kruijff et al, 2007;Kozierok et al, 2021;Krishnaswamy and Pustejovsky, 2021).…”