Remote XR Studies: Exploring Three Key Challenges of Remote XR Experimentation

Ratcliffe, Jack; Soave, Francesco; Hoover, Melynda; Ortega, Francisco R.; Bryan-Kinns, Nick; Tokarchuk, Laurissa; Farkhatdinov, Ildar

doi:10.1145/3411763.3442472

Cited by 13 publications

(6 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is important to note that remote iVR data collection methods pose specific challenges that can affect different aspects of studies focusing on behavioral outcome of users, such as the one performed in this paper. In laboratory settings, researchers can assess many different aspects of users such as physiological data, skin conductance, heart-rate, blood pressure, Electroencephalography, eye-tracking, positional tracking data, and more, that can prove to be vital and revealing in understanding the behavioral outcomes of users (Ratcliffe et al, 2021b). Some of such data can be extracted from some of the available HMDs in the market (e.g., HTC Vive Pro Eye 12 , HP Omnicept Reverb 13 ), but these devices are often more expensive and not really of interest to the population of users who purchase iVR devices for entertainment purposes.…”

Section: Resultsmentioning

confidence: 99%

Remote iVR for Nutrition Education: From Design to Evaluation

Sajjadi

Edwards

Zhao

et al. 2022

Front. Comput. Sci.

View full text Add to dashboard Cite

While different crowdsourcing platforms promote remote data collection, experiments in the immersive Virtual Reality (iVR) research community are predominantly performed in person. The COVID-19 pandemic, however, has forced researchers in different disciplines, including iVR, to seriously consider remote studies. In this paper, we present a remote study using the Immersive Virtual Alimentation and Nutrition (IVAN) application, designed to educate users about food-energy density and portion size control. We report on the results of a remote experiment with 45 users using the IVAN app. In IVAN, users actively construct knowledge about energy density by manipulating virtual food items, and explore the concept of portion size control through hypothesis testing and assembling virtual meals in iVR. To explore the feasibility of conducting remote iVR studies using an interactive health-related application for nutrition education, two conditions were devised (interactive vs. passive). The results demonstrate the feasibility of conducting remote iVR studies using health-related applications. Furthermore, the results also indicate that regardless of level of interactivity learners significantly improved their knowledge about portion size control after using the IVAN (p < 0.0001). Adding interactivity, however, suggests that the perceived learning experience of users could be partially affected. Learners reported significantly higher scores for immediacy of control in the interactive condition compared to those in the passive condition (p < 0.05). This study demonstrates the feasibility of conducting an unsupervised remote iVR experiment using a complex and interactive health-related iVR app.

show abstract

Section: Resultsmentioning

confidence: 99%

Remote iVR for Nutrition Education: From Design to Evaluation

Sajjadi

Edwards

Zhao

et al. 2022

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…To this end, it is possible to devise new options for remote XR studies by selecting options from the attributes in the leaf nodes of the taxonomy for each dimension and sub-dimension. We note that although the Finally, we note that this taxonomy describes the most universal attributes which we found relevant in literature (MacKenzie, 2013;Steed et al, 2016Steed et al, , 2020Moran, 2020;Moran and Pernice, 2020;Wiberg et al, 2020;Ratcliffe et al, 2021b;Schmidt et al, 2021;Spittle et al, 2021) and through our implementation of several remote studies detailed above (plus other ongoing and/or unpublished studies). The taxonomy is likely best viewed as a starting point, and we do not claim that it is fully complete.…”

Section: Proposed Xr Remote Studies Taxonomymentioning

confidence: 76%

“…Of particular recent interest is so-called remote studies; user studies that resemble traditional lab-based studies, but are deployed remotely with little (or sometimes no) experimenter intervention. Remote studies have become more prominent in response to the COVID-19 pandemic (Steed et al, 2016(Steed et al, , 2020Moran, 2020;Moran and Pernice, 2020;Wiberg et al, 2020;Ratcliffe et al, 2021b). HCI research is inherently interdisciplinary (MacKenzie, 2013) and can benefit from surveying how remote studies are employed in related fields (Frippiat and Marquis, 2010;Chaudhuri, 2020).…”

mentioning

confidence: 99%

The reality of remote extended reality research: Practical case studies and taxonomy

Kroma

Grinyer

Scavarelli

et al. 2022

Front. Comput. Sci.

View full text Add to dashboard Cite

Remote user studies—those where the experimenter and participant are not physically located together—offer challenges and opportunities in HCI research in general, and extended reality (XR) research specifically. The COVID-19 pandemic has forced this form of research to overcome a long period of unprecedented circumstances. However, this experience has produced a lot of lessons learned that should be shared. We propose guidelines based on findings from a set of six remote virtual reality studies, by analyzing participants' and researchers' feedback. These studies ranged from one-session types to longitudinal ones and spanned a variety of subjects such as cybersickness, selection tasks, and visual search. In this paper, we offer a big-picture summary of how we conducted these studies, our research design considerations, our findings in these case studies, and what worked well and what did not in different scenarios. Additionally, we propose a taxonomy for devising such studies in a systematic and easy-to-follow manner. We argue that the XR community should move from theoretical proposals and thought pieces to testing and sharing practical data-informed proposals and guidelines.

show abstract

“…For remote data collection, Ma et al (2018) identified challenges in crowd-sourced remote immersive VR experiments, e.g., the heterogeneity of hardware, participants trying to trick the experiment, and participant pool size and composition. The Human Computer Interaction (HCI) community has documented several reproducibility challenges, e.g., Ratcliffe et al (2021) collected experiences and challenges of remote experimentation in HCI research, while Echtler and Häußler (2018) documented and discusses the reproducibility crisis and the extent of use of open source and open science concepts in HCI. Feger et al (2019) glances over the roles and challenges of HCI on research reproducibility, e.g., the incentives for making research reproducible and the potential role of gamification mechanisms.…”

Section: Introductionmentioning

confidence: 99%

Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments

Aguilar¹,

Gath-Morad²,

Grübel³

et al. 2022

Preprint

View full text Add to dashboard Cite

A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians and engineers) and may require many resources (e.g. cloud infrastructure, specialized equipment). Even though researchers strive to document experiments accurately, this process is often lacking, making it hard to reproduce them. Moreover, when it is necessary to create a similar experiment, very often we end up "reinventing the wheel" as it is easier to start from scratch than trying to reuse existing work, thus losing valuable embedded best practices and previous experiences. In behavioral studies this has contributed to the reproducibility crisis. To tackle this challenge, we propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code to provision, deploy, manage, and analyze it is provided. To this end we define the Experiments as Code concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with a simple desktop VR experiment that showcases the benefits of its "as code" representation, i.e., reproducibility, auditability, debuggability, reusability, & scalability.

show abstract

Remote XR Studies: Exploring Three Key Challenges of Remote XR Experimentation

Cited by 13 publications

References 5 publications

Remote iVR for Nutrition Education: From Design to Evaluation

Remote iVR for Nutrition Education: From Design to Evaluation

The reality of remote extended reality research: Practical case studies and taxonomy

Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments

Contact Info

Product

Resources

About