Video Conference System in Mixed Reality Using a Hololens

Abstract

The mixed reality conference system proposed in this paper is a robust, real-time video conference application software that makes up for the simple interaction and lack of immersion and realism of traditional video conference, which realizes the entire process of holographic video conference from client to cloud to the client. This paper mainly focuses on designing and implementing a video conference system based on AI segmentation technology and mixed reality. Several mixed reality conference system components are discussed, including data collection, data transmission, processing, and mixed reality presentation. The data layer is mainly used for data collection, integration, and video and audio codecs. The network layer uses Web-RTC to realize peer-to-peer data communication. The data processing layer is the core part of the system, mainly for human video matting and human-computer interaction, which is the key to realizing mixed reality conferences and improving the interactive experience. The presentation layer explicitly includes the login interface of the mixed reality conference system, the presentation of real-time matting of human subjects, and the presentation objects. With the mixed reality conference system, conference participants in different places can see each other in real-time in their mixed reality scene and share presentation content and 3D models based on mixed reality technology to have a more interactive and immersive experience.

Design and Implementation of Two Immersive Audio and Video Communication Systems Based on Virtual Reality

“…This method could display the real image of people in a virtual space, as well as the body information in real time. However, the participants in the images still wore a head display, which may reduce the effectiveness of communication.A novel network video conference system based on MR display devices and AI segmentation technology was proposed in [22]. This was a robust, real-time video conference software application that made up for the simple interaction and lack of immersion and realism of traditional video conference, allowing users to interact in a conference in a new way.…”

XR CUBE: Multi-Sensory Intelligent Interaction Device for Extended Reality Application

“…Thirdly, with the development of artificial intelligence technology, some studies have implemented gesture recognition [14], natural language recognition [15], AI segmentation technology [16] and other techniques based on machine learning [12], biosignals, and other methods to achieve a more diverse and realistic interaction experience in XR environments. Ratcliffe et al [13] summarized the results of 30 surveys from 46 XR researchers, and then determined the portability and reproducibility of leveraging XR devices' built-in data collection capabilities (e.g., hand, gaze tracking).…”

Reconstruction of Industrial and Historical Heritage for Cultural Enrichment Using Virtual and Augmented Reality

“…This allows for a more immersive and interactive experience, as the user can interact with virtual objects in a way that is similar to how they would interact with physical objects in the real world. MR applications and technologies have become quite popular nowadays [31,32]. They are used in many areas (e.g., games [33], entertainment [34], education [35], assisted living [36], psychology [37], health and rehabilitation [38,39], etc.).…”

“…The MR applications and technologies are becoming increasingly popular, according to [31,32,35]. They are used in a variety of contexts (e.g., games, entertainment, education, etc.).…”