Parallel Real-Time Tracking and 3D Reconstruction with TBB for Intelligent Control and Smart Sensing Framework

Abstract

Recent advancements in aircraft controllers paired with increasingly flexible aircraft designs create the need for adaptive and intelligent control systems. To correctly capture the motion of a flexible aircraft wing and provide feedback to the controller, a large number of states (nodes along the span) must be monitored in real-time. Visual sensing methods carry the promise of flexibility needed for this type of smart sensing and control. However, visual sensing requires capturing and tracking keypoint features (marker tracking), while detecting thereof from a feature-rich image can be a computationally intensive task. The computational effort significantly increases with image size or when an image stereo pair is used to find matching keypoints. In this study, a parallel approach is presented with Threading Building Blocks (TBB), using sub-matrix computations, for extraction of corresponding keypoints from an image-stereo pair, and triangulation with the Direct Linear Transform (DLT) method to reconstruct the 3D position of the object in space. Additional robustness is investigated by implementing a Kalman filter for tracking prediction during the domain transition between the sub-matrices. Furthermore, a flexible simulation framework is set up for smart sensing with a coupled unsteady aeroservoelastic model of a 3D wing and a visual model to test the method for intelligent control feedback in a simulation environment. The methodology is tested in a laboratory environment with a stereo camera setup, and in a virtual environment, where the virtual camera parameters are reconstructed to meet a stereo setup. The proposed approach aims to advance the state-of-the-art in smart sensing, particularly in the context of real-time state estimation of aeroelastic structures and control feedback. The parallel approach shows a significant improvement of speed and efficiency, allowing real-time computation from a live image stream at 50 fps.

Design and Development of a Seamless Smart Morphing Wing Using Distributed Trailing Edge Camber Morphing for Active Control

“…II, were integrated into the morphing wing. These are composed of state-of-the-art : (i) piezoelectric pressure sensors, (ii) embedded fibre-optic sensors, (iii) seamless morphing control and (iv) and visual tracking system for control feedback [14,15].…”

“…• Sensors in the wing allow real-time measurements of (i) global shape of the wing with fibre optic sensors, (ii) flow separation and control e ectiveness of the morphing surface, using piezoelectric sensors (iii) position feedback of the morphing surfaces using high-speed cameras installed at the root to feed to the controller [14,15],.…”

Adaptive State Estimation and Real-Time tracking of Aeroelastic Wings with Augmented Kalman filter and Kernelized Correlation Filter

“…7. Further details regarding the 3D reconstruction can be found in a previous study by Mkhoyan et al [9] and [10].…”

“…Code development, testing and assessment was done using standard Dell Optiplex 7400 and 2.3 GHz Intel Core i5 16G MacBook and the Jetson TX2. A part of the code, dataset and tools developed in the scope of the study are available under the repositories tmkhoyan/adaptiveClusteringTracker [8,10] and tmkhoyan/parallelTrackingTBB [9,20].…”

“…To obtain the reconstructed displacements, the camera setup was calibrated and the corresponding KCF-EKF point pairs from two image streams triangulated with the DLT method. The details of the triangulation approach are given in [9,20]. The obtained 3D coordinates were then transformed to align the laser vibrometer measurement.…”