Fully Onboard AI-Powered Human-Drone Pose Estimation on Ultralow-Power Autonomous Flying Nano-UAVs

Abstract

Many emerging applications of nano-sized unmanned aerial vehicles (UAVs), with a few cm 2 form-factor, revolve around safely interacting with humans in complex scenarios, for example, monitoring their activities or looking after people needing care. Such sophisticated autonomous functionality must be achieved while dealing with severe constraints in payload, battery, and power budget (∼100 mW). In this work, we attack a complex task going from perception to control: to estimate and maintain the nano-UAV's relative 3D pose with respect to a person while they freely move in the environment -a task that, to the best of our knowledge, has never previously been targeted with fully onboard computation on a nano-sized UAV. Our approach is centered around a novel vision-based deep neural network (DNN), called PULP-Frontnet, designed for deployment on top of a parallel ultra-low-power (PULP) processor aboard a nano-UAV. We present a vertically integrated approach starting from the DNN model design, training, and dataset augmentation down to 8-bit quantization and deployment in-field. PULP-Frontnet can operate in real-time (up to 135 frame/s), consuming less than 87 mW for processing at peak throughput and down to 0.43 mJ/frame in the most energy-efficient operating point. Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a tiny 27-grams Crazyflie 2.1 nano-UAV. Compared against an ideal sensing setup, onboard pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41 cm, ideal: 26 cm) and angular (onboard: 3.7 • , ideal: 4.1 • ). We publicly release videos and the source code of our work.

Neural Control and Online Learning for Speed Adaptation of Unmanned Aerial Vehicles

“…These control parameters can be the positions, flying speeds, and acceleration, depending on the system and mission goal. Similarly, in deep learning (Kaufmann et al, 2019 ; Varshney et al, 2019 ; Palossi et al, 2022 ) and RL (Shin et al, 2019 ; Singla et al, 2019 ), these parameters are optimized through the learning process, which typically requires a large number of dataset and numerous iterations. In comparison to existing state-of-the-art UAV control techniques ( Table 2 ), our neural predictive control technique does not require a UAV dynamic model, a simulated environment for learning, nor does it require the collection of a training dataset in advance.…”

“…However, the system design of this technique should be considered, including sensor installation, sensor detection range, and parameter tuning, to ensure the effectiveness of the control system. For automatic parameter tuning or optimization, various machine learning methods have been applied including deep learning (Palossi et al, 2019 , 2022 ; Varshney et al, 2019 ), reinforcement learning (RL) (Shin et al, 2019 ; Wang et al, 2019 ; Lin et al, 2020 ), and an evolutionary algorithm (EA) (Fu et al, 2018 ; Yazid et al, 2019 ). Although they have become more popular in recent years, their learning processes are typically time-consuming and computationally expensive, requiring a large amount of data and multiple learning trials or iterations.…”

A Deep Learning-Based Face Mask Detector for Autonomous Nano-Drones (Student Abstract)

“…The COVID19 outbreak has shown the importance of quick social reactions, including monitoring the use of protective face masks among the population. With their sub-10cm size and a few tens of grams in weight, nano-sized unmanned aerial vehicles (UAVs) are the ideal candidates to safely fly in human proximity and perform ubiquitous visual perception tasks (Palossi et al 2021), such as the mask detection use case we address in this work (see Figure 1). However, enabling high-level sensing capabilities on nano-UAVs, i.e., without relying on off-board computational resources, is hindered by their small form factor, which limits the on-board computational/memory/sensory resources to the class of ultra-low-power devices.…”

“…However, enabling high-level sensing capabilities on nano-UAVs, i.e., without relying on off-board computational resources, is hindered by their small form factor, which limits the on-board computational/memory/sensory resources to the class of ultra-low-power devices. Convolutional neural networks (CNNs) are essential for visual pattern recognition tasks: with reduced memory and computational requirements, these models are a perfect fit for full deployment on resource-constrained embedded platforms, such as those found on small nano-UAVs (Palossi et al 2021).…”

Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes

“…Deep Learning (DL) models can enhance the "smartness' of many edge systems, such as drones [1], wearables [2], and smart assistants [3]. However, the traditional paradigm based on offloading Deep Neural Networks (DNNs) execution to the cloud has several limitations in terms of latency, energy efficiency, and privacy, as it relies on continuous data exchange over the network, often through an unpredictable, unreliable and energy hungry wireless channel [4].…”