Telemedicine System Based on Mixed Reality and Cognitive Speech Service Technologies

Lin, Pei-Jung; Tsai, Bo-Chang; Tsai, Yung-Wei

doi:10.1109/ecbios54627.2022.9944986

Cited by 6 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It will build a medical system based on a hybrid reality and use technology from the metaverse in order to provide more online interactive services. In further research, the goal is to find a way to combine a virtual reality rehabilitation training system with telemedicine so that it can provide a more all-encompassing service [41].…”

Section: Literature Reviewmentioning

confidence: 99%

An Overview of Speech-To-Text Conversion

Aggarwal

2023

CIML

View full text Add to dashboard Cite

As a result of developments in science and technology, an automatic speech-to-text (STT) conversion system has been available. This system converts spoken words into text that can be read visually. People with trouble hearing may use this technology to communicate in other ways, including understanding voice communication and being able to follow directions using their visual abilities. There are instances when seeing something is more powerful than listening to something, particularly in long-distance communication; thus, speech-to-text conversion is crucial in situations like these. One of the fascinating developments to occur in the twenty-first century is the advent of machine learning. It has evolved from its roots in neurology studies conducted in the 1940s into something like artificial intelligence humans have created. Neural networks, a collection of complex structures, are the basis of machine learning. When combined with optimization techniques, these networks mimic the behaviour of neurons in the human brain and allow a computer to learn from its experiences. Here we explore one of many potential uses for such structures - the analysis of vocal performance in an original study. In particular, we dissect voice recognition systems to determine their inner workings.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

An Overview of Speech-To-Text Conversion

Aggarwal

2023

CIML

View full text Add to dashboard Cite

show abstract

“…Regarding the usage of XR technology in telemedicine applications, we can mention the solution proposed in [ 3 ], where an MR teleconsultation system was realized. Azure Kinect DK is exploited on the patient side, while a doctor uses Microsoft HoloLens 2, which allows manipulation of 3D organ models and medical images.…”

Section: Related Workmentioning

confidence: 99%

Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation

Šarić,

Russo,

Kraljević

et al. 2023

Sensors

View full text Add to dashboard Cite

Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.

show abstract

MazeMind: Exploring the Effects of Hand Gestures and Eye Gazing on Cognitive Load and Task Efficiency in an Augmented Reality Environment

Sun,

Liao

2024

Design Computing and Cognition’24

View full text Add to dashboard Cite

Telemedicine System Based on Mixed Reality and Cognitive Speech Service Technologies

Cited by 6 publications

References 13 publications

An Overview of Speech-To-Text Conversion

An Overview of Speech-To-Text Conversion

Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation

MazeMind: Exploring the Effects of Hand Gestures and Eye Gazing on Cognitive Load and Task Efficiency in an Augmented Reality Environment

Contact Info

Product

Resources

About