INC-Hg: An Intelligent Collaborative Haptic-Gripper Virtual Reality System

Abstract

Collaborative Virtual Environments (CVE) have shown potential to be an effective social skill training platform for children with Autism Spectrum Disorders (ASD) to learn and practice collaborative and communication skills through peer interactions. However, most existing CVE systems require that appropriately matched partners be available at the same time to promote interaction, which limits their applicability to some community settings due to scheduling constraints. A second shortcoming of these more naturalistic peer-based designs is the intensive resources required to manually code the unrestricted conversations that occurred during the peer-based interactions. To preserve the benefits of CVE-based platforms and mitigate some of the resource limitations related to peer availability, we developed an Intelligent Collaborative Haptic-Gripper System (INC-Hg). This system provides an intelligent agent partner who can understand, communicate, and haptically interact with the user, without requiring the presence of another human peer. The INC-Hg operates in real time and thus is able to perform collaborative training tasks at any time and at the user's pace. INC-Hg can also record the real-time data regarding spoken language and task performance, thereby greatly reducing the resource burden of communication and interaction performance analysis. A preliminary usability study with 10 participants with ASD (ages 8–12 years) indicated that the system could classify the participant's utterances into five classes with an accuracy of 70.34%, which suggested the potential of INC-Hg to automatically recognize and analyze conversational content. The results also indicated high accuracies of the agent to initiate a conversation (97.56%) and respond to the participants (86.52%), suggesting the capability of the agent to conduct proper conversations with the participants. Compared to the results of human-to-human collaborative tasks, the human-to-agent mode achieved higher average collaborative operation ratio (61% compared to 40%) and comparable average frequencies for Initiations and Responses among the participants with ASD. These results offer preliminary support as well as areas of improvement regarding the agent's ability to respond to participants, work with participants to complete tasks, engage in back-and-forth conversations, and support the potential of the agent to be a useful partner for individuals with ASD completing CVE tasks.

Design of a Desktop Virtual Reality-Based Collaborative Activities Simulator (ViRCAS) to Support Teamwork in Workplace Settings for Autistic Adults

“…The haptic device was the least favorite device compared to the other devices, which indicated that ease of use was more important to the participants than immersive interaction since participants were least familiar with the use of the haptic device. Existing studies that explored the use of haptic devices in VR-based interactions have shown that with practice, haptic devices can be well accepted by participants [23], [28]. Therefore, for future studies involving teamwork skills training, we will use either the gamepad or keyboard and mouse, while haptic device could be used in studies that examine fine motor skills training.…”

“…Effective teamwork requires collaboration among individuals working together, making collaboration an important indicator of teamwork performance [27]. A collaborative virtual environment (CVE) extends the benefits of conventional VR technology by supporting multi-user interaction within the shared virtual space, allowing users to naturally communicate with each other [28], potentially increasing generalizability of learned skills to the real world. Several recent CVE-based interactions have been promising.…”

“…Previous CVE-based studies also explored nonverbal aspects of social communication such as joint attention skills [32], joint action skills [33], and imitation skills [34] where improvements in these skills were observed after completing multiple training sessions in laboratory settings. More recently, researchers found promising results by combining social communication skills and motor skills training for autistic children where they reported increased performance in both domains [28]. However, to our knowledge, there are no CVE-based studies that target social interactions within the employment landscape, which can differ from everyday social interactions.…”

Serious Game com Técnicas de Treinamento Motor Fino como Tecnologia Assistiva para encorajar a Colaboração de Crianças com TEA

“…Habilidades motoras finas são habilidades que exigem o uso dos músculos para pequenas ac ¸ões como agarrar, manipular e alcanc ¸ar objetos [Arshad et al 2020]. A partir disso, pesquisas na literatura indicam que o uso de jogos digitais impacta positivamente as habilidades motoras finas de crianc ¸as com TEA ao estimular melhorias significativas no desempenho de tarefas individuais e colaborativas e ao promover a comunicac ¸ão social e a colaborac ¸ão [Zhao et al 2021].…”

“…Visando apoiar os processos terapêuticos, foram identificadas tecnologias na literatura que usam de jogos e AVs 3D para treinamento de habilidades motoras, além da colaborac ¸ão e a comunicac ¸ão, com foco em crianc ¸as com TEA [Zhao et al 2021], [Giraud et al 2021], [Amat et al 2021]. Tarefas motoras finas na forma de jogos de computador podem proporcionar recompensas ou consequências consistentes e em tempo real para as respostas, o que pode incentivar o envolvimento e reter a atenc ¸ão das crianc ¸as [Zhao et al 2018].…”