The Φ-Sat-1 Mission: The First On-Board Deep Neural Network Demonstrator for Satellite Earth Observation

Abstract

Artificial intelligence (AI) is paving the way for a new era of algorithms focusing directly on the information contained in the data, autonomously extracting relevant features for a given application. While the initial paradigm was to have these applications run by a server hosted processor, recent advances in microelectronics provide hardware accelerators with an efficient ratio between computation and energy consumption, enabling the implementation of AI algorithms "at the edge." In this way only the meaningful and useful data are transmitted to the end-user, minimizing the required data bandwidth, and reducing the latency with respect to the cloud computing model. In recent years, European Space Agency (ESA) is promoting the development of disruptive innovative technologies on-board earth observation (EO) missions. In this field, the most advanced experiment to date is the -sat-1, which has demonstrated the potential of artificial intelligence (AI) as a reliable and accurate tool for cloud detection on-board a hyperspectral imaging mission. The activities involved included demonstrating the robustness of the Intel Movidius Myriad 2 hardware accelerator against ionizing radiation, developing a Cloudscout segmentation neural network (NN), run on Myriad 2, to identify, classify, and eventually discard on-board the cloudy images, and assessing the innovative Hyperscout-2 hyperspectral sensor. This mission represents the first official attempt to successfully run an AI deep convolutional NN (CNN) directly inferencing on a dedicated accelerator on-board a satellite, opening the way for a new era of discovery and commercial applications driven by the deployment of on-board AI.

Self-Supervised Vessel Detection from Low Resolution Satellite Imagery

“…On-orbit processing is an emerging field in the setting of deep-learning based earth observation problems. Initial implementations, such as the CloudScout neural network on-board ESA's Φ-Sat 1, have yielded promising results [6,15]. Implementing neural networks on-orbit provides several advantages to small satellite operations, including a heavily reduced communications bandwidth [16], improved scalability, and the capability for each satellite to be tuned and re-trained with its captured raw data [17], tailoring an algorithm to a specific end device.…”

“…Unfortunately, due to a variety of factors relating to the resolution and source platforms that the datasets were built from, these datasets were not suitable for use in this work. This is an issue seen in other papers in the field [4,6], which has lead to authors creating their own novel datasets for algorithms; an approach which can also be seen implemented here.…”

“…4. First, the created inputs were normalized between the values of 0 and 1 (a technique used to improve convergence time [6]), and were then fed through into the encoder model using the Keras library input layer, hosted within the TensorFlow Python libraries.…”

“…These systems allow for effective, near real-time surveillance of Earth's oceans, with suitable spatiotemporal coverage. Traditionally, these factors often came with a compromise of reduced on-board computing ability and poor sensor resolution, however recent advances in the qualification of COTS components for space has seen complex algorithms being introduced onto more capable flight hardware [4,6].…”

Automatic Deployment of Convolutional Neural Networks on FPGA for Spaceborne Remote Sensing Application

“…Therefore, MicroSats and NanoSats cannot adopt expensive, large-scale FPGAs as computation devices, which means DSPs and other hardware resources are precious to them. Meanwhile, spaceborne remote sensing platforms not only perform CNN-based image processing, but also perform image preprocessing, such as radiation correction and image dehazing, to improve the performance of CNN-based image processing [24]. Some FPGA-based studies showed that image preprocessing requires a lot of hardware resources to implement [25,26].…”