Accelerating real-time embedded scene labeling with convolutional networks

Abstract

Today there is a clear trend towards deploying advanced computer vision (CV) systems in a growing number of application scenarios with strong real-time and power constraints. Brain-inspired algorithms capable of achieving recordbreaking results combined with embedded vision systems are the best candidate for the future of CV and video systems due to their flexibility and high accuracy in the area of image understanding. In this paper, we present an optimized convolutional network implementation suitable for real-time scene labeling on embedded platforms. We show that our algorithm can achieve up to 96 GOp/s, running on the Nvidia Tegra K1 embedded SoC. We present experimental results, compare them to the state-of-the-art, and demonstrate that for scene labeling our approach achieves a 1.5x improvement in throughput when compared to a modern desktop CPU at a power budget of only 11 W.

fpgaConvNet: A Framework for Mapping Convolutional Neural Networks on FPGAs

“…These include three well-known ConvNets for computer vision applications with different computational loads, namely the Convolutional Face Finder (CFF) [12], LeNet-5 [13] and MPCNN [14]. We also implemented two ConvNets from existing FPGA works [4][7] for scene labelling and sign recognition which we denote CNP and Sign Recognition respectively, and one ConvNet for scene labelling from an embedded GPU work [15].…”

“…To evaluate the quality in terms of performance of the design point that is selected by our framework, we compare fpgaConvNet with the FPGA works of [4] and [7] and the embedded GPU work of [15]. Fig.…”

“…In these cases, large images increase the computation-to-reconfiguration time ratio, but at the expense of higher on-chip memory requirements for line buffering inside sliding window units. Finally, we compared the fpgaConvNet's performance with the embedded GPU work presented in [15] for the same scene labelling ConvNet. As shown in table III, our framework reaches 16.75% of the Tegra K1's sustained performance as reported in [15], clocked at 1/8 of the frequency and overpasses its performance efficiency in GOp/s/W by 1.05×.…”

Origami

“…While we keep our implementation runtime-configurable to a large extent, we use the ConvNet presented in [10] as a reference for performance evaluation. It has three stages and we assume input images of size 240 × 320.…”

“…On the Tegra K1 up to 96 GOp/s can be achieved, with 76 GOp/s being achieved with an actual ConvNet. On both platforms an energy-efficiency of about 7 GOp/s/W considering the power of the entire platform and 14.4 GOp/s/W with differential power measurements can be obtained [10].…”