Using Images Rendered by PBRT to Train Faster R-CNN for UAV Detection

Abstract

Deep neural networks, such as Faster R-CNN, have been widely used in object detection. However, deep neural networks usually require a large-scale dataset to achieve desirable performance. For the specific application, UAV detection, training data is extremely limited in practice. Since annotating plenty of UAV images manually can be very resource intensive and time consuming, instead, we use PBRT to render a large number of photorealistic UAV images of high variation within a reasonable time. Using PBRT ensures the realism of rendered images, which means they are indistinguishable from real photographs to some extent. Trained with our rendered images, the Faster R-CNN has an AP of 80.69% on manually annotated UAV images test set, much higher than the one only trained with COCO 2014 dataset and PASCAL VOC 2012 dataset (43.36%). Moreover, our rendered image dataset contains not only bounding boxes of all UAVs, but also locations of some important parts of UAVs and locations of all pixels covered by UAVs, which can be used for more complicated application, such as mask detection or keypoint detection.

Machine Learning-Based Drone Detection and Classification: State-of-the-Art in Research

“…To mitigate this situation, some authors made use of transfer learning rather than starting from scratch. Other research work such as [32] employed dedicated software to generate synthetic images to increase the number of samples in the dataset. Other techniques for enlarging the dataset that could be used in the future include data augmentation and the utilization of generative models such as generative adversarial network (GAN) for creating artificial data which are similar to the original real data.…”

“…Peng et al in [32] addressed the issue of limited visual data for UAVs by creating their own artificial images. They used Physically Based Rendering Toolkit (PBRT) to generate photorealistic UAV images.…”

Drone Model Identification by Convolutional Neural Network from Video Stream

“…It is done by overlaying 3d drone models on top of random backgrounds. Another method [17] improves on this by using a Physically Based Rendering Toolkit to render photorealistic images of drones.…”

Real-Time and Accurate Drone Detection in a Video with a Static Background

“…Drone detection based on visual data (image or video) can be performed using handcrafted feature-based methods [ 8 , 21 , 22 ] and deep learning-based [ 6 , 23 , 24 , 25 ] algorithms. Handcrafted feature-based methods are based on traditional machine learning algorithms by using traditional descriptors such as scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), Haar, local binary pattern (LBP), deformable parts model (DPM), and generic Fourier descriptor (GFD) that provide low-level handcrafted features (edges, drops, blobs, and color information) and classical classifiers (support vector machine (SVM), AdaBoost)), whereas the second category relies on the learned features using two-stage (region-based convolutional neural network (R-CNN), Fast R-CNN, Faster R-CNN, and Mask R-CNN) and single-stage (single shot detector (SSD), RetinaNet, and you only look once (YOLO)) deep object detectors.…”

“…The experiment results showed that, despite training in synthetic data, the proposed system worked well on realistic images of drones against a complex background. Peng et al [ 25 ] used the physical rendering instrumentation tool (PBRT) to solve the problem of limited visual data by creating photorealistic images of UAVs. The authors developed a large-scale training set of 60,480 rendered images, choosing different positions and orientations of UAVs, 3D models, external materials, internal and external camera characteristics, environmental maps, and the post-processing of rendered images.…”