Visual Odometry for Pixel Processor Arrays

Bose, Laurie; Chen, Jianing; Carey, Stephen J.; Dudek, Piotr; Mayol-Cuevas, Walterio

doi:10.1109/iccv.2017.493

Cited by 31 publications

(33 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Image computations are carried-out in these processors, and only sparse outputs are transmitted off the sensor device. These devices have been demonstrated to offer unique advantages in applications such as keypoint extraction [6], depth from focus [17], or visual odometry [3]. In this work, we consider implementations of CNNs on such devices.…”

Section: Introductionmentioning

confidence: 99%

A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

Bose

Dudek

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

We present a convolutional neural network implementation for pixel processor array (PPA) sensors. PPA hardware consists of a fine-grained array of general-purpose processing elements, each capable of light capture, data storage, program execution, and communication with neighboring elements. This allows images to be stored and manipulated directly at the point of light capture, rather than having to transfer images to external processing hardware. Our CNN approach divides this array up into 4x4 blocks of processing elements, essentially trading-off image resolution for increased local memory capacity per 4x4 "pixel". We implement parallel operations for image addition, subtraction and bit-shifting images in this 4x4 block format. Using these components we formulate how to perform ternary weight convolutions upon these images, compactly store results of such convolutions, perform max-pooling, and transfer the resulting sub-sampled data to an attached micro-controller. We train ternary weight filter CNNs for digit recognition and a simple tracking task, and demonstrate inference of these networks upon the SCAMP5 PPA system. This work represents a first step towards embedding neural network processing capability directly onto the focal plane of a sensor.

show abstract

Section: Introductionmentioning

confidence: 99%

A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

Bose

Dudek

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The SCAMP-5 PPA camera attached to the underside of the vehicle as shown in Figure 2 is used to produce an estimate of the vehicle's motion. The motion estimate is made by building upon image alignment techniques first introduced in our previous paper [11]. The image alignment process used involves determining the transformation (consisting of translational, scaling and rotational components) to apply to the latest acquired image in-order to best align it with the current key-frame.…”

Section: Methods a Odometrymentioning

confidence: 99%

“…This allows robust image alignment to still be performed under rapid camera motion not possible when using a standard camera. A full description of this image alignment process is given in [11].…”

Section: Methods a Odometrymentioning

confidence: 99%

“…In this work the motion of the quadrotor MAV is estimated from the rate at which the scene moves across the images captured by the PPA sensor. This scene tracking is achieved by an image alignment process, building upon previous work in [11], where the current image is aligned against a previous acquired key-frame, in a manner exploiting the parallel nature of the PPA architecture. The output of this scale-less visual odometry method is converted into position and velocity using the MAV's height sensor, as discussed in Section III-A.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Perspective Correcting Visual Odometry for Agile MAVs using a Pixel Processor Array

Greatwood

Bose

Richardson

et al. 2018

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

This paper presents a visual odometry approach using a Pixel Processor Array (PPA) camera, specifically, the SCAMP-5 vision chip. In this device, each pixel is capable of storing data and performing computation, enabling a variety of computer vision tasks to be carried out directly upon the sensor itself. In this work the PPA performs HDR edge detection, perspective correction and image alignment based odometry, allowing the position and heading of a MAV to be tracked at several hundred frames per second. We evaluate our PPA based approach by direct comparison with a motion capture system for a variety of trajectories. These include rapid accelerations that would incur significant motion blur at low frame rates, and lighting conditions that would typically lead to under or over exposure of image detail. Such challenging conditions would often lead to unusable images when relying on traditional image sensors.

show abstract

“…New directions are being followed to explore the performance of FPSPs in real-world robotic and virtual reality applications. These directions include 1) 4-DOF camera tracking [63], and 2) automatic filter kernel code generation as well as Viola-Jones [64] face detection [65]. The key concept behind these works with FPSP is the fact that FPSP is able to report sum of intensity values of all (or a selection of) pixels in just one clock cycle.…”

Section: Advanced Sensorsmentioning

confidence: 99%

Navigating the Landscape for Real-Time Localization and Mapping for Robotics and Virtual and Augmented Reality

et al. 2018

View full text Add to dashboard Cite

Visual understanding of 3D environments in realtime, at low power, is a huge computational challenge. Often referred to as SLAM (Simultaneous Localisation and Mapping), it is central to applications spanning domestic and industrial robotics, autonomous vehicles, virtual and augmented reality. This paper describes the results of a major research effort to assemble the algorithms, architectures, tools, and systems software needed to enable delivery of SLAM, by supporting applications specialists in selecting and configuring the appropriate algorithm and the appropriate hardware, and compilation pathway, to meet their performance, accuracy, and energy consumption goals. The major contributions we present are (1) tools and methodology for systematic quantitative evaluation of SLAM algorithms, (2) automated, machine-learning-guided exploration of the algorithmic and implementation design space with respect to multiple objectives, (3) end-to-end simulation tools to enable optimisation of heterogeneous, accelerated architectures for the specific algorithmic requirements of the various SLAM algorithmic approaches, and (4) tools for delivering, where appropriate, accelerated, adaptive SLAM solutions in a managed, JIT-compiled, adaptive runtime context. Performance EvaluationRuntime Architecture Compiler and Algorithm Design Space Exploration -Machine Learning Fig. 1: The objective of the paper is to create a pipeline that aligns computer vision requirements with hardware capabilities. The paper's focus is on three layers: algorithms, compiler and runtime, and architecture. The goal is to develop a system that allows us to achieve power and energy efficiency, speed and runtime improvement, and accuracy/robustness at each layer and also holistically through design space exploration and machine learning techniques.

show abstract

Visual Odometry for Pixel Processor Arrays

Cited by 31 publications

References 17 publications

A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

Perspective Correcting Visual Odometry for Agile MAVs using a Pixel Processor Array

Navigating the Landscape for Real-Time Localization and Mapping for Robotics and Virtual and Augmented Reality

Contact Info

Product

Resources

About