Hardware-in-the-loop simulation with dynamic partial FPGA reconfiguration applied to computer vision in ROS-based UAV

Moréac, Erwan; Abdali, El Mehdi; Berry, François; Heller, Dominique; Diguet, Jean-Philippe

doi:10.1109/rsp51120.2020.9244863

Cited by 11 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The flight controller was implemented as an overlay based on the PYNQ project. In 2020, Moreac et al [23] proposed FPGA based computer vision for drones by hardware-in-the-loop simulation. Finally in 2021 at our research group, Kövari [24] demonstrated the use of FPGA accelerated deep learning inference for autonomous drone navigation, also based on PYNQ.…”

Section: Related Workmentioning

confidence: 99%

MPSoC4Drones: An Open Framework for ROS2, PX4, and FPGA Integration

Nyboe

Malle

Ebeid

2022

2022 International Conference on Unmanned Aircraft Systems (ICUAS)

View full text Add to dashboard Cite

Autonomous drones are facing a tough efficiency challenge due to limitations on the utilized processing hardware units. Among these limitations is the tradeoff between fast computing and low power consumption; between functional complexity and flight time. Recent progressions point to FPGAs for accelerating heavy processing. In this work, we present the MPSoC4Drones Framework; a novel framework for organizing FPGA-design and OS build projects. The framework combines tools for creating bootable images for the Ultra96-V2 board.We show how MPSoC4Drones organizes the build, combining the latest well-known tools for research and industrial drone development, Ubuntu 20.04, PX4 autopilot, and ROS2 middleware. We validate the framework through a computationally intensive deep learning use case implemented in the MPSoC4Drones framework. We show the superior throughput and low power consumption of FPGA processing compared to classical CPU and GPU approaches. Finally, we offer the full framework as open-source.

show abstract

Section: Related Workmentioning

confidence: 99%

MPSoC4Drones: An Open Framework for ROS2, PX4, and FPGA Integration

Nyboe

Malle

Ebeid

2022

2022 International Conference on Unmanned Aircraft Systems (ICUAS)

View full text Add to dashboard Cite

show abstract

“…Additionally, none of the methods have been demonstrated to run onboard a UAV in real-time. Because of their power efficiency and high throughput capability, FPGAs have been used on UAVs as flight controllers [40], neural network accelerators [41], [42], and generally deployed for their reconfigurability [43], [44]. NASA's Mars UAV [45] uses FPGAs for flight control, fault tolerance, and as IO and communications hubs.…”

Section: Related Workmentioning

confidence: 99%

“…NASA's Mars UAV [45] uses FPGAs for flight control, fault tolerance, and as IO and communications hubs. FPGAs have also shown great promise for embedded, real-time implementations of advanced algorithms such as linear and nonlinear Model Predictive Control (MPC) [46], [47], [48] and various computer vision algorithms [44], [49]. Ladig et al [50] show an FPGA-accelerated perception system that assists a pilot in yaw-alignment of a drone relative to detected lines.…”

Section: Related Workmentioning

confidence: 99%

Onboard Powerline Perception System for UAVs Using mmWave Radar and FPGA-Accelerated Vision

2022

View full text Add to dashboard Cite

Autonomous Unmanned Aerial Vehicle (UAV) interactions with powerlines, such as close-up inspections for fault detection or grasping and landing for recharging, require advanced onboard perception capabilities. To solve such tasks, the UAV must be equipped with perception abilities that allow it to navigate between powerlines and safely approach specific cables of interest. A perception system with such capabilities requires state-of-the-art sensor technologies and data processing while still being subject to the limited hardware and energy resources of the UAV. In this paper, we present an advanced embedded system based on the cutting-edge Multiprocessing System-on-Chip (MPSoC) for onboard UAV powerline perception. Our platform consists of a mmWave radar and an RGB camera with data processing carried out on the MPSoC, covering both CPU and Field-Programmable Gate Array (FPGA) computations. Following hardware-software co-design methodology, the heavy image processing tasks are accelerated in the FPGA and fused with computationally light mmWave data on the CPU, facilitating pose-estimation of the power lines. Utilizing the open-source autonomy frameworks PX4 and ROS2, we demonstrate integration of the system with onboard path planning based on the estimated cable positions. The robustness of the detection and pose-estimation methods have been demonstrated in several tests performed both in simulated and real-world powerline environments. The results show that our proposed perception system allows the UAV to safely navigate in close proximity to powerlines, by perceiving more individual cables at longer distances compared to previous work, while remaining lightweight, power-efficient, and low-cost.

show abstract

“…As an example of this proposed new architecture, we have started the development and test of dynamically reconfigurable hardware. We use ROS and interface it with the FPGA to take advantage of partially and dynamically reconfigurable hardware [ 124 ]. Such a complex system can only be developed by combining efforts and contributions from the research community.…”

Section: Addressing the Computational Gap For High Levels Of Autonmentioning

confidence: 99%

Embedded Computation Architectures for Autonomy in Unmanned Aircraft Systems (UAS)

Mejias

Diguet²,

Dezan

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

This paper addresses the challenge of embedded computing resources required by future autonomous Unmanned Aircraft Systems (UAS). Based on an analysis of the required onboard functions that will lead to higher levels of autonomy, we look at most common UAS tasks to first propose a classification of UAS tasks considering categories such as flight, navigation, safety, mission and executing entities such as human, offline machine, embedded system. We then analyse how a given combination of tasks can lead to higher levels of autonomy by defining an autonomy level. We link UAS applications, the tasks required by those applications, the autonomy level and the implications on computing resources to achieve that autonomy level. We provide insights on how to define a given autonomy level for a given application based on a number of tasks. Our study relies on the state-of-the-art hardware and software implementations of the most common tasks currently used by UAS, also expected tasks according to the nature of their future missions. We conclude that current computing architectures are unlikely to meet the autonomy requirements of future UAS. Our proposed approach is based on dynamically reconfigurable hardware that offers benefits in computational performance and energy usage. We believe that UAS designers must now consider the embedded system as a masterpiece of the system.

show abstract

Hardware-in-the-loop simulation with dynamic partial FPGA reconfiguration applied to computer vision in ROS-based UAV

Cited by 11 publications

References 18 publications

MPSoC4Drones: An Open Framework for ROS2, PX4, and FPGA Integration

MPSoC4Drones: An Open Framework for ROS2, PX4, and FPGA Integration

Onboard Powerline Perception System for UAVs Using mmWave Radar and FPGA-Accelerated Vision

Embedded Computation Architectures for Autonomy in Unmanned Aircraft Systems (UAS)

Contact Info

Product

Resources

About