Background COVID-19 restrictions severely curtailed empirical endeavors that involved in-person interaction, such as usability testing sessions for technology development. Researchers and developers found themselves using web-based moderation for usability testing. Skilled remote moderators and observers are fundamental in this approach. However, to date, more empirical work is needed that captures the perceptions and support needs of moderators and observers in testing situations. Objective The aim of this paper was to identify remote moderator and observer participant experiences and their use of certain tools to capture feedback of users as they interact with the web browser application. Methods This research is part of a broader study on an educational web browser application for nursing students to learn perspective taking and enhance their perceptual understanding of a dialogue partner’s thoughts and feelings. The broader study used a quantitative and think-aloud qualitative problem-discovery usability study design. This case study explored written accounts of the remote moderator and observer participants regarding their roles, experiences, and reactions to the testing protocol and their suggestions for improved techniques and strategies for conducting remote usability testing. Content analysis was used to analyze participants’ experiences in the usability testing sessions. Results We collected data from 1 remote moderator and 2 remote observers. Five themes were identified: dealing with personal stressors, dealing with user anxiety, maintaining social presence, ethical response to the study protocol, and communication during sessions. The participants offered recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. Conclusions This study’s findings contribute to a growing body of endeavors to understand human-computer interaction and its impact on remote moderator and observer roles. As technology rapidly advances, more remote usability testing will occur where the knowledge gleaned in this study can have an impact. Recommendations based on moderator and observer participant perspectives identify the need for more evidence-informed training materials for their roles that focus on web-based interpersonal communication skills, execution of user testing protocols, troubleshooting technology and test user issues, proficiency in web conferencing platforms, behavior analysis and feedback technologies, and time management.

Background The prepandemic period saw a rise in web-based teaching. However, web-based tools for teaching the essential clinical skill of cognitive empathy (also known as perspective taking) remain limited. More of these tools are needed and require testing for ease of use and understanding by students. Objective This study aimed to evaluate the usability of the In Your Shoes web-based empathy training portal application for students using quantitative and qualitative methods. Methods This 3-phase formative usability study used a mixed methods design. In mid-2021, we conducted a remote observation of student participants interacting with our portal application. Their qualitative reflections were captured, followed by data analysis and iterative design refinements of the application. Overall, 8 third- and fourth-year nursing students from an undergraduate baccalaureate program at a Canadian university, in the western province of Manitoba, were included in this study. Participants in phases 1 and 2 were remotely observed by 3 research personnel while engaged in predefined tasks. In phase 3, two student participants were asked to use the application as they liked in their own environments, after which a video-recorded exit interview with a think-aloud process was conducted as participants responded to the System Usability Scale. We calculated descriptive statistics and performed content analysis to analyze the results. Results This small study included 8 students with a range of technology skills. Usability themes were based on participants’ comments on the application’s appearance, content, navigation, and functionality. The biggest issues that participants experienced were with navigating the application’s “tagging” features during video analysis and the length of educational material. We also observed variations in 2 participants’ system usability scores in phase 3. This may be because of their different comfort levels with technology; however, additional research is required. We made iterative refinements to our prototype application (eg, added pop-up messages and provided a narrated video on the application’s “tagging” function) based on participant feedback. Conclusions With increasing engagement in web-based teaching, technology has become an essential medium for receiving health care education. We developed a novel prototype application as a supplemental classroom tool to foster students’ self-directed learning of empathy. This study provided direction for refinements to optimize the usability of and satisfaction with this innovative application. Qualitative feedback revealed favorable input toward learning perspective taking place on the web and helpful recommendations for improving user experiences with the application. We could not fully assess the application’s key functions owing to the COVID-19 protocols. Thus, our next step is to obtain feedback from a larger sample of student users, whose experiences performing “live” video capture, annotation, and analysis will be more authentic and wholesome with the refined application. We discuss our findings in relation to research on nursing education, perspective taking, and adaptive e-learning.

BACKGROUND COVID 2019 restrictions severely curtailed empirical endeavors that involved in-person human interaction. The pandemic also stimulated our team to embrace technology in a two-fold manner. First, we created novel technology to help us to overcome pandemic restrictions in teaching empathic communication in the traditional classroom. A web-based training portal was created for users to learn empathy in an accessible, compelling, self-directed, and interactive online environment. Second, we harnessed technology to engage in remote usability testing and data collection with prospective users of our training portal. In developing our protocol, we discovered gaps in the literature on moderator and silent observer roles and experiences in conducting remote usability testing. OBJECTIVE The aim of this paper is to share our remote moderator and silent observer experiences and their use of certain tools to capture user feedback and experiences with the app. METHODS The larger-scale project employed a quantitative and think-aloud qualitative problem-discovery usability study design. Three trained research assistants collected and utilized user feedback from eight users who were asked to complete tasks in three sessions. Each research assistant had assigned roles and were asked to qualitatively describe their roles, experiences and reactions to the usability testing protocol, and suggestions for improved techniques and strategies for conducting remote usability testing. RESULTS Major results are described in relation to the research assistant experiences with the study protocol followed by recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. CONCLUSIONS Our findings highlighted that as we move towards greater remote usability testing, we also need more comprehensive understanding of human-computer interaction and its impact on usability testing outcomes. Our team realized that accessible comprehensive web-conferencing platform to conduct remote sessions is not sufficient. Lead moderator and silent observers offered their insights and recommendations for the ongoing creation and testing of training materials for their respective roles with a focus on: online interpersonal communication skills, conducting user testing protocols, troubleshooting technology and test user issues, proficiency in web-conferencing plus behavior analysis and feedback technologies, and time management. CLINICALTRIAL None.