Visual Odometry Using Pixel Processor Arrays for Unmanned Aerial Systems in GPS Denied Environments

Abstract

Environments in which Global Positioning Systems (GPS), or more generally Global Navigation Satellite System (GNSS), signals are denied or degraded pose problems for the guidance, navigation, and control of autonomous systems. This can make operating in hostile GNSS-Impaired environments, such as indoors, or in urban and natural canyons, impossible or extremely difficult. Pixel Processor Array (PPA) cameras-in conjunction with other on-board sensors-can be used to address this problem, aiding in tracking, localization, and control. In this paper we demonstrate the use of a PPA device-the SCAMP vision chip-combining perception and compute capabilities on the same device for aiding in real-time navigation and control of aerial robots. A PPA consists of an array of Processing Elements (PEs), each of which features light capture, processing, and storage capabilities. This allows various image processing tasks to be efficiently performed directly on the sensor itself. Within this paper we demonstrate visual odometry and target identification running concurrently on-board a single PPA vision chip at a combined frequency in the region of 400 Hz. Results from outdoor multirotor test flights are given along with comparisons against baseline GPS results. The SCAMP PPA's High Dynamic Range (HDR) and ability to run multiple algorithms at adaptive rates makes the sensor well suited for addressing outdoor flight of small UAS in GNSS challenging or denied environments. HDR allows operation to continue during the transition from indoor to outdoor environments, and in other situations where there are significant variations in light levels. Additionally, the PPA only needs to output specific information such as the optic flow and target position, rather than having to output entire images. This significantly reduces the bandwidth required for communication between the sensor and on-board flight computer, enabling high frame rate, low power operation.

Method of Real-Time Video Stream Synchronization in the Working Environment of an Apple Orchard

“…Another approach [10] is to use visual odometry, which is a technique for estimating the motion of a vehicle by analyzing the changes in the images captured by its camera. In other work [11], each UAV is equipped with a visual odometry system that estimates its motion in real time. The estimated motion is then used to align the UAV video streams.…”

Sensor-level computer vision with pixel processor arrays for agile robots

“…Another key capability of a PPA is the potential to perform more than one function or competence on a single sensor, primarily enabled by the on-sensor processing power available. In (87) we combine visual odometry with HDR imaging and target detection to reset the VO estimates for a drone surveying a target area. This functionality would be appropriate in a GPS denied or limited availability setting.…”

On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays

“…This helps to reduce the key factors on embedded systems such as latency, energy consumption, and bandwidth, as data movements are optimized and redundant information is eliminated close to its source. The SCAMP-5d vision system (Dudek, 2004 ; Carey et al, 2013 ) is a general-purpose programmable massively parallel PPA vision system that has been recently demonstrated in robotic tasks (Greatwood et al, 2017 , 2018 ; McConville et al, 2020 ; Fan et al, 2021 ; Liu et al, 2021a ) and computer vision (Bose et al, 2017 ; Chen et al, 2017 ; Martel et al, 2020 ). Figure 1 , illustrates how the SCAMP's Processing Element (PE) uses a photosensor to convert light into an analog signal which is then directly processed in the adjacent arithmetic logic unit (ALU) and analog (AREG) and binary (DREG) registers.…”