Establishing multimodal telepresence sessions using the Session Initiation Protocol (SIP) and advanced haptic codecs

Abstract

In telepresence and telemanipulation systems, multimodal data is exchanged over a network allowing humans to experience and to operate in remote or inaccessible environments. To operate over the global Internet and connect to multiple telepresence systems, a flexible framework for initiating, handling and terminating Internetbased telerobotic sessions becomes necessary. In this work, we explore the use of standard Internet session and transport protocols in the context of telerobotic applications.Session Initiation Protocol (SIP) is widely used to handle multimedia teleconference sessions with audio, video or text, and provides many services advantageous for establishing connections between heterogeneous haptic interfaces and telerobotic systems. We apply the session paradigm to the creation and negotiation of haptic telepresence sessions and propose to extend this framework to work with the haptic modality. The notion of a "haptic codec" is introduced for transforming haptic data into a common format, applying data reduction or compression techniques and implementing teleoperation control architectures. The use of the Real-Time Transport Protocol (RTP) is explored for transport of teleoperation data. Finally, a prototype and demonstrator system is presented for evaluation of the proposed framework.

Toward Haptic Communications Over the 5G Tactile Internet

“…Again, based on the client-server scheme, in [54] haptic communication is achieved by employing the Session Initiation Protocol (SIP) on the application layer in order to establish a teleoperation session and to manage haptic transport streams that use Real-Time Transport Protocol (RTP) which encapsulates the haptic data in UDP packets. In this case, SIP allows for having an abstraction layer in order to incorporate encoded data in the packets using a haptic codec.…”

Immersive Interconnected Virtual and Augmented Reality: A 5G and IoT Perspective

“…Several attempts have been made in this direction. In [62], haptic communication is achieved by employing the Session Initiation Protocol (SIP) to establish a teleoperation session and manage haptic transport streams. The multimodal communication framework of PAHCP [63] synchronizes data by using the Network Time Protocol (NTP) while graphics and haptic data are transmitted with Virtual Network Connection (VNC) and PAHCP respectively.…”

Augmented reality and haptic interfaces for robot‐assisted surgery

“…The University of Washington and Technischen Universität München also collaborated to establish multimodal telepresence, using a session initiation protocol. They used standard Internet session and transport protocols to share audio, video and haptic information . In this paper, we describe an interface specifically designed to enable novel haptic feedback capabilities.…”