FPGA-Based CNN for Real-Time UAV Tracking and Detection

Abstract

Neural networks (NNs) are now being extensively utilized in various artificial intelligence platforms specifically in the area of image classification and real-time object tracking. We propose a novel design to address the problem of real-time unmanned aerial vehicle (UAV) monitoring and detection using a Zynq UltraScale FPGA-based convolutional neural network (CNN). The biggest challenge while implementing real-time algorithms on FPGAs is the limited DSP hardware resources available on FPGA platforms. Our proposed design overcomes the challenge of autonomous real-time UAV detection and tracking using a Xilinx’s Zynq UltraScale XCZU9EG system on a chip (SoC) platform. Our proposed design explores and provides a solution for overcoming the challenge of limited floating-point resources while maintaining real-time performance. The solution consists of two modules: UAV tracking module and neural network–based UAV detection module. The tracking module uses our novel background-differencing algorithm, while the UAV detection is based on a modified CNN algorithm, designed to give the maximum field-programmable gate array (FPGA) performance. These two modules are designed to complement each other and enabled simultaneously to provide an enhanced real-time UAV detection for any given video input. The proposed system has been tested on real-life flying UAVs, achieving an accuracy of 82%, running at the full frame rate of the input camera for both tracking and neural network (NN) detection, achieving similar performance than an equivalent deep learning processor unit (DPU) with UltraScale FPGA-based HD video and tracking implementation but with lower resource utilization as shown by our results.

Joint UAV deployment, SF placement, and collaborative task scheduling in heterogeneous multi‐UAV‐empowered edge intelligence

“…Therefore, Constraints (4), (7), and ( 10) are all satisfied. Finally, based on the matrix V , A, and O, the maximum overall revenue G max and the suboptimal solution of SF placement and collaborative task scheduling X can be obtained (in steps [5][6][7][8][9][10][11][12][13][14][15]. The initialization of V = (v i j ) K ×K and O = (o i j ) K ×K is shown in Algorithm 2.…”

“…is better than the old individual L u(i ) and will be retained. Otherwise, the old individual L u(i ) is kept (in steps [10][11][12]. When the evolution of the population ends, we choose the individual with the maximum overall revenue as the sub-optimal solution of L u (in step 16).…”

“…We consider a general computation model for the UAV that includes two parts: CPU and CP resources, as shown in Figure 2. The former includes several CPU cores of the same type, while the latter refers to a set of FPGAs, which assist CPUs in performing AI workload, like FPGA‐based DNN accelerator [10, 11]. The connection between the CPU cores and the CPs can be established dynamically utilizing internal bus (e.g.…”

“…The AI CPs, such as Field Programmable Gate Array (FPGA), GPU, and Application Specific Integrated Circuit (ASIC), handle the AI workload and leave the rest of the task to the CPU [6,7]. FPGA has been treated as a promising solution to supplant conventional processors for performing computation-intensive tasks [8][9][10][11], such as deep neural network (DNN)-based image recognition, on the UAV platforms without violating size, weight, and power constraints inherent to UAV design [12][13][14][15]. Unless otherwise stated, we adopt FPGAs as the CPs in the following analysis.…”

On-board drone classification with Deep Learning and System-on-Chip implementation

“…Concerning implementation performance, it depends on the processing hardware/software architecture. Despite of the classical CPU/GPU implementation, recently Field Programmable Gate Arrays (FPGAs) based System-on-Chip (SoC) have become a highly attractive platform for implementing CNNs in real-time, as FPGAs are generally more energy-efficient [18], [19]. Thanks to programmable logic inside FPGA devices, it is possible to design a custom hardware accelerator that can be tailored on the task of a CNN.…”