Moderator and observer experiences and decision-making: Remote usability testing of a web-based empathy training portal (Preprint)

Abstract

BACKGROUND COVID 2019 restrictions severely curtailed empirical endeavors that involved in-person human interaction. The pandemic also stimulated our team to embrace technology in a two-fold manner. First, we created novel technology to help us to overcome pandemic restrictions in teaching empathic communication in the traditional classroom. A web-based training portal was created for users to learn empathy in an accessible, compelling, self-directed, and interactive online environment. Second, we harnessed technology to engage in remote usability testing and data collection with prospective users of our training portal. In developing our protocol, we discovered gaps in the literature on moderator and silent observer roles and experiences in conducting remote usability testing. OBJECTIVE The aim of this paper is to share our remote moderator and silent observer experiences and their use of certain tools to capture user feedback and experiences with the app. METHODS The larger-scale project employed a quantitative and think-aloud qualitative problem-discovery usability study design. Three trained research assistants collected and utilized user feedback from eight users who were asked to complete tasks in three sessions. Each research assistant had assigned roles and were asked to qualitatively describe their roles, experiences and reactions to the usability testing protocol, and suggestions for improved techniques and strategies for conducting remote usability testing. RESULTS Major results are described in relation to the research assistant experiences with the study protocol followed by recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. CONCLUSIONS Our findings highlighted that as we move towards greater remote usability testing, we also need more comprehensive understanding of human-computer interaction and its impact on usability testing outcomes. Our team realized that accessible comprehensive web-conferencing platform to conduct remote sessions is not sufficient. Lead moderator and silent observers offered their insights and recommendations for the ongoing creation and testing of training materials for their respective roles with a focus on: online interpersonal communication skills, conducting user testing protocols, troubleshooting technology and test user issues, proficiency in web-conferencing plus behavior analysis and feedback technologies, and time management. CLINICALTRIAL None.