Applications of Machine Learning and Monocular Vision for Autonomous On-Orbit Proximity Operations

Dhamani, Nihal; Martin, Gavin; Schubert, Carson; Singh, Pratyush; Hatten, Noble; Akella, Maruthi R.

doi:10.2514/6.2020-1376

Cited by 5 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it is uniquely difficult to simulate realistic orbital scenes due to the diversity of lighting conditions, vehicle orientations, and image noise present in the space environment. A major takeaway from our prior work on the NASA Seeker mission [9] is that models trained on a sub-par synthetic image dataset, both in terms of fidelity and diversity, can struggle to generalize well to the real environment, a finding in line with other works like [10].…”

Section: Introductionmentioning

confidence: 62%

“…The Seeker Vision system, detailed in [9] and flown on NASA JSC's Seeker 1 Cubesat [11], utilized a convolutional neural network to identify and localize the Cygnus spacecraft in monocular imagery. Development and performance of this model was hindered by both challenges described previously: an inefficient dataset curation, training, and evaluation system along with sub-par synthetic imagery.…”

Section: Motivationmentioning

confidence: 99%

See 1 more Smart Citation

A Pipeline for Vision-Based On-Orbit Proximity Operations Using Deep Learning and Synthetic Imagery

Schubert

Black

Fonseka

et al. 2021

2021 IEEE Aerospace Conference (50100)

Self Cite

View full text Add to dashboard Cite

Deep learning has become the gold standard for image processing over the past decade. Simultaneously, we have seen growing interest in orbital activities such as satellite servicing and debris removal that depend on proximity operations between spacecraft. However, two key challenges currently pose a major barrier to the use of deep learning for vision-based onorbit proximity operations. Firstly, efficient implementation of these techniques relies on an effective system for model development that streamlines data curation, training, and evaluation. Secondly, a scarcity of labeled training data (images of a target spacecraft) hinders creation of robust deep learning models. This paper presents an open-source deep learning pipeline, developed specifically for on-orbit visual navigation applications, that addresses these challenges. The core of our work consists of two custom software tools built on top of a cloud architecture that interconnects all stages of the model development process. The first tool leverages Blender, an open-source 3D graphics toolset, to generate labeled synthetic training data with configurable model poses (positions and orientations), lighting conditions, backgrounds, and commonly observed in-space image aberrations. The second tool is a plugin-based framework for effective dataset curation and model training; it provides common functionality like metadata generation and remote storage access to all projects while giving complete independence to project-specific code. Time-consuming, graphics-intensive processes such as synthetic image generation and model training run on cloud-based computational resources which scale to any scope and budget and allow development of even the largest datasets and models from any machine. The presented system has been used in the Texas Spacecraft Laboratory with marked benefits in development speed and quality. Remote development, scalable compute, and automatic organization of data and artifacts have dramatically decreased iteration time while increasing reproducibility and system comprehension. Diverse, high-fidelity synthetic images that more closely replicate the real environment have improved model performance against real-world data. These results demonstrate that the presented pipeline offers tangible benefits to the application of deep learning for vision-based on-orbit proximity operations.

show abstract

Section: Introductionmentioning

confidence: 62%

Section: Motivationmentioning

confidence: 99%

A Pipeline for Vision-Based On-Orbit Proximity Operations Using Deep Learning and Synthetic Imagery

Schubert

Black

Fonseka

et al. 2021

2021 IEEE Aerospace Conference (50100)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Yong et al [16] and Tao et al [17] achieved space object recognition using a multilayer convolutional neural network based on LeNet and AlexNet separately. Dhamani et al [18] selected the MobileNet V1 Single Shot Detector (SSD) architecture for space object detection.…”

Section: Related Workmentioning

confidence: 99%

Visible and Infrared Image Fusion-Based Image Quality Enhancement with Applications to Space Debris On-Orbit Surveillance

Tao

Cao

Ding

et al. 2022

International Journal of Aerospace Engineering

View full text Add to dashboard Cite

The increasing amount of space debris in recent years has greatly threatened space operation. In order to ensure the safety level of spacecraft, space debris perception via on-orbit visual sensors has become a promising solution. However, the perception capability of visual sensors largely depends on illumination, which tends to be insufficient in dark environments. Since the images captured by visible and infrared sensors are highly complementary in dark environments, a convolutional sparse representation-based visible and infrared image fusion algorithm is proposed in this paper to expand the applicability of visual sensors. In particular, the local contrast measure is applied to obtain the refined weight map for fusing the base layers, which is more robust in a dark space environment. The algorithm can settle two significant problems in space debris surveillance, namely, improving the signal-noise ratio in a noise space environment and preserving more detailed information in a dark space environment. A space debris dataset containing registered visible and infrared images has been purposely created and used for algorithm evaluation. Experimental results demonstrate that the proposed method in this paper is effective for enhancing image qualities and can achieve favorable effects compared to other state-of-the-art algorithms.

show abstract

“…To benchmark the pose estimation model, we use the Intel Joule 570x single-board computer. It has been used previously in a CNN-based visual navigation system onboard a 3U CubeSat, 13 establishing its viability as flight-ready commercial off-the-shelf (COTS) hardware. The computational capabilities of the Joule are severely limited in comparison to typical ground-based hardware such as a laptop or smartphone.…”

Section: Hardware Performancementioning

confidence: 99%

“…In addition, we present a thorough benchmark of the system's performance on an Intel Joule 570x single-board computer, the same type which flew on the NASA Johnson Space Center (JSC) Seeker CubeSat mission in September 2019. 13 We first provide a survey of related work. Then, we describe the details of our pose estimation architecture, as well as our synthetic data generation scheme using the Northrop Grumman Enhanced Cygnus vehicle as our target spacecraft.…”

Section: Introductionmentioning

confidence: 99%

Real-Time, Flight-Ready, Non-Cooperative Spacecraft Pose Estimation Using Monocular Imagery

Black,

Shankar,

Fonseka

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

A key requirement for autonomous on-orbit proximity operations is the estimation of a target spacecraft's relative pose (position and orientation). It is desirable to employ monocular cameras for this problem due to their low cost, weight, and power requirements. This work presents a novel convolutional neural network (CNN)-based monocular pose estimation system that achieves state-of-the-art accuracy with low computational demand. In combination with a Blender-based synthetic data generation scheme, the system demonstrates the ability to generalize from purely synthetic training data to real in-space imagery of the Northrop Grumman Enhanced Cygnus spacecraft. Additionally, the system achieves realtime performance on low-power flight-like hardware.

show abstract

Applications of Machine Learning and Monocular Vision for Autonomous On-Orbit Proximity Operations

Cited by 5 publications

References 8 publications

A Pipeline for Vision-Based On-Orbit Proximity Operations Using Deep Learning and Synthetic Imagery

A Pipeline for Vision-Based On-Orbit Proximity Operations Using Deep Learning and Synthetic Imagery

Visible and Infrared Image Fusion-Based Image Quality Enhancement with Applications to Space Debris On-Orbit Surveillance

Real-Time, Flight-Ready, Non-Cooperative Spacecraft Pose Estimation Using Monocular Imagery

Contact Info

Product

Resources

About