NeurAll: Towards a Unified Visual Perception Model for Automated Driving

Abstract

Convolutional Neural Networks (CNNs) are successfully used for the important automotive visual perception tasks including object recognition, motion and depth estimation, visual SLAM, etc. However, these tasks are typically independently explored and modeled. In this paper, we propose a joint multi-task network design for learning several tasks simultaneously. Our main motivation is the computational efficiency achieved by sharing the expensive initial convolutional layers between all tasks. Indeed, the main bottleneck in automated driving systems is the limited processing power available on deployment hardware. There is also some evidence for other benefits in improving accuracy for some tasks and easing development effort. It also offers scalability to add more tasks leveraging existing features and achieving better generalization. We survey various CNN based solutions for visual perception tasks in automated driving. Then we propose a unified CNN model for the important tasks and discuss several advanced optimization and architecture design techniques to improve the baseline model. The paper is partly review and partly positional with demonstration of several preliminary results promising for future research. We first demonstrate results of multi-stream learning and auxiliary learning which are important ingredients to scale to a large multi-task model. Finally, we implement a two-stream three-task network which performs better in many cases compared to their corresponding single-task models, while maintaining network size.

WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving

“…Multi-task: Autonomous driving has various vision tasks and most of the work has been focused on solving individual tasks independently. However, there is a recent trend [29,50,48,6] to solve tasks using a single multi-task model to enable efficient reuse of encoder features and also provide regularization while learning multiple tasks. However, in these cases, only the encoder is shared and there is no synergy among decoders.…”

Adversarial Attacks on Multi-task Visual Perception for Autonomous Driving

“…However, in real-life situations, multi-task learning is adopted to solve several tasks at once [13]. Accordingly, multi-task networks are used to leverage the shared knowledge among tasks, leading to better performance, reduced storage, and faster inference [14]. Moreover, it is shown that when models are trained on multiple tasks at once, they become more robust to adversarial attacks on individual tasks [15].…”

CompositeTasking: Understanding Images by Spatial Composition of Tasks

“…Intuitively, different image understanding tasks offer complementary information for scene understanding and reasoning [24,53,2,62,57,61,11,50]. Therefore, networks that can perform multiple visual tasks on the same image are of very high interest [10,34,22,51,55].…”