Visual Inertial Odometry At the Edge: A Hardware-Software Co-design Approach for Ultra-low Latency and Power

Mandal, Dipan Kumar; Jandhyala, Srivatsava; Omer, Om Ji; Kalsi, Gurpreet S.; George, Biji; Neela, Gopi; Rethinagiri, Santhosh Kumar; Subramoney, Sreenivas; Hacking, Lance; Radford, Jim; Jones, Eagle; Kuttanna, B.; Wang, Hong

doi:10.23919/date.2019.8714921

Cited by 13 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A lower pose update latency was required for the estimation of the 6-DoF pose of the HMD to achieve a latency of less than 20 ms for a motion-sickness-free immersive experience. The noisy data obtained from the camera were analyzed by a maximum likelihood estimator to estimate the system dynamics or the state in a typical VIO processing pipeline [ 79 ]. There are some VIO systems whose functioning is similar to an extended Kalman filter (EKF) [ 80 ] to update their states and decide the best pose estimate available.…”

Section: Ar/vr Hardware and Software Requirementsmentioning

confidence: 99%

“…The best pose is decided immediately upon receiving the sensory inputs, including the image frame or IMU sample, without waiting for an optimal solution via local or global optimization based on past estimates. The acceleration of EKF-based monocular VIO [ 81 ] was modeled in [ 79 ], by a flexible, scalable, and power-efficient micro-architecture. This enabled direct processing of IMU samples without pre-integration to obtain an ultra-low latency and accurate instantaneous pose updates.…”

Section: Ar/vr Hardware and Software Requirementsmentioning

confidence: 99%

See 1 more Smart Citation

Towards an Evolved Immersive Experience: Exploring 5G- and Beyond-Enabled Ultra-Low-Latency Communications for Augmented and Virtual Reality

Hazarika

Rahmati

2023

Sensors

View full text Add to dashboard Cite

Augmented reality and virtual reality technologies are witnessing an evolutionary change in the 5G and Beyond (5GB) network due to their promising ability to enable an immersive and interactive environment by coupling the virtual world with the real one. However, the requirement of low-latency connectivity, which is defined as the end-to-end delay between the action and the reaction, is very crucial to leverage these technologies for a high-quality immersive experience. This paper provides a comprehensive survey and detailed insight into various advantageous approaches from the hardware and software perspectives, as well as the integration of 5G technology, towards 5GB, in enabling a low-latency environment for AR and VR applications. The contribution of 5GB systems as an outcome of several cutting-edge technologies, such as massive multiple-input, multiple-output (mMIMO) and millimeter wave (mmWave), along with the utilization of artificial intelligence (AI) and machine learning (ML) techniques towards an ultra-low-latency communication system, is also discussed in this paper. The potential of using a visible-light communications (VLC)-guided beam through a learning algorithm for a futuristic, evolved immersive experience of augmented and virtual reality with the ultra-low-latency transmission of multi-sensory tracking information with an optimal scheduling policy is discussed in this paper.

show abstract

Section: Ar/vr Hardware and Software Requirementsmentioning

confidence: 99%

Section: Ar/vr Hardware and Software Requirementsmentioning

confidence: 99%

Towards an Evolved Immersive Experience: Exploring 5G- and Beyond-Enabled Ultra-Low-Latency Communications for Augmented and Virtual Reality

Hazarika

Rahmati

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Hence, the optimal solutions are used in the proposed hardware system. As for the second step, the best scales of level 1 to level 4 of the hardware parallel image pyramid along with the PSNR value of each level of image is presented in Table II [15] and various feature-based VIO systems including [10], [12], [14], and [17]. Although different dataset scenarios are used in these methods, the image size is similar.…”

Section: ) Hardware Implemented Parallel Image Pyramidmentioning

confidence: 99%

“…Alike [12], [10], and [16] where an additional IMU is employed, [17] also built a VIO system in a HW/SW co-3 design manner using FAST features. However, unlike [12], [10] and [16], the processing of the IMU as well as the synchronization between the camera and the IMU were taken over by software, while the rest of the system was managed by hardware.…”

Section: Introductionmentioning

confidence: 99%

“…Albeit many advantages brought by a HW/SW codesign, the difference of operational frequencies between software and hardware sometimes lowers the overall performances and scales up the development complexity. For instance, in [17], its use of FPGA and ARM respectively operate at 70 MHz and 1.2 GHz. As a consequence, this paper builds on our previous work [18] and carries out a multiple master-slave HW/SW co-design stereo VO architecture using SIFT features on an Altera FPGA board.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multiple Master-Slave FPGA Architecture of a Stereo Visual Odometry

2021

View full text Add to dashboard Cite

Estimating relative camera pose is the key problem of visual odometry (VO). To achieve better efficiency, sparse key-points are usually relied on for the estimation. Yet, feature extraction and matching are still computationally demanding, hindering the overall VO from real-time processing. Exploiting the superiorities of an FPGA in terms of high efficiency, low power consuming, and low cost, this paper proposes a multiple master-slave FPGA architecture for an SIFT-based stereo VO. The masterslave design enables high reconfiguration for the data throughputs among various modules. These modules include SIFT, matching, pose estimation, and their corresponding controllers. In the SIFT module, hardware implemented image pyramid is proposed, where scales are determined off-line via a minimization approach. Local linear exhausted search (LES) matching is considered for both the stereo and the frame matching. In the pose estimation module, a novel hardware design of deriving closest orthogonal matrix for 3D-3D correspondences of relative pose estimation is proposed. Experimental results show that 33.2 fps can be achieved using KITTI dataset without the need of large amount of hardware resources. The proposed reconfigurable design also facilitates its expansions of adopting CCD cameras as well as developing SLAM and other applications.

show abstract

Hardware implementation of SLAM algorithms: a survey on implementation approaches and platforms

2022

View full text Add to dashboard Cite

Visual Inertial Odometry At the Edge: A Hardware-Software Co-design Approach for Ultra-low Latency and Power

Cited by 13 publications

References 8 publications

Towards an Evolved Immersive Experience: Exploring 5G- and Beyond-Enabled Ultra-Low-Latency Communications for Augmented and Virtual Reality

Towards an Evolved Immersive Experience: Exploring 5G- and Beyond-Enabled Ultra-Low-Latency Communications for Augmented and Virtual Reality

Multiple Master-Slave FPGA Architecture of a Stereo Visual Odometry

Hardware implementation of SLAM algorithms: a survey on implementation approaches and platforms

Contact Info

Product

Resources

About