$$\text{C}^{3}\text{Net}$$: end-to-end deep learning for efficient real-time visual active camera control

Abstract

The need for automated real-time visual systems in applications such as smart camera surveillance, smart environments, and drones necessitates the improvement of methods for visual active monitoring and control. Traditionally, the active monitoring task has been handled through a pipeline of modules such as detection, filtering, and control. However, such methods are difficult to jointly optimize and tune their various parameters for real-time processing in resource constraint systems. In this paper a deep Convolutional Camera Controller Neural Network is proposed to go directly from visual information to camera movement to provide an efficient solution to the active vision problem. It is trained end-to-end without bounding box annotations to control a camera and follow multiple targets from raw pixel values. Evaluation through both a simulation framework and real experimental setup, indicate that the proposed solution is robust to varying conditions and able to achieve better monitoring performance than traditional approaches both in terms of number of targets monitored as well as in effective monitoring time. The advantage of the proposed approach is that it is computationally less demanding and can run at over 10 FPS ( ∼ 4× speedup) on an embedded smart camera providing a practical and affordable solution to real-time active monitoring.

Eagle: End-to-end Deep Reinforcement Learning based Autonomous Control of PTZ Cameras

“…The object occurrence probability identifies the less frequent objects of interest, and a rule-based controller modifies the PTZ parameters. Using neural network object detectors makes it infeasible to deploy this pipeline on embedded camera platforms [14]. Further, multi-stage information flow necessitates fine-tuning of each stage, and its performance is impacted by errors in each stage [14,16].…”

“…Using neural network object detectors makes it infeasible to deploy this pipeline on embedded camera platforms [14]. Further, multi-stage information flow necessitates fine-tuning of each stage, and its performance is impacted by errors in each stage [14,16]. E.g., the object detector's errors impact the tracking performance [14].…”

“…Relative_location+control [12,14]: Steps 1 6 7 9 . A neural network is trained using supervised machine learning to output the relative location of the object of interest in captured images.…”

“…Expensive fine-tuning: The multiple stages suffer from performance bottlenecks as it is non-trivial to tune each step. For example, tuning the parameters of the Kalman filter requires expert domain knowledge and incurs trial-and-error [14,16]. Fine-tuning neural object detectors requires bounding box labels [8,16].…”

“…2. Real-time deployment on resource-constrained platforms: Even lightweight object detectors (such as YOLO [2,14]) with several millions of network parameters are too complex for embedded camera platforms. This makes it infeasible to run multi-stage PTZ control algorithms in real-time on platforms [1,14] with memory and computation constraints.…”