Satellite Image Time Series Classification With Pixel-Set Encoders and Temporal Self-Attention

Garnot, Vivien Sainte Fare; Landrieu, Loïc; Giordano, Sébastien; Chehata, Nesrine

doi:10.1109/cvpr42600.2020.01234

Cited by 78 publications

(84 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning has been applied using convolutional neural networks (CNNs) for handling the temporal dimension [21,35]; recurrent neural networks (RNNs)-like models [36,37], including long short term memory (LSTM) [33,38] or gated recurrent unit (GRU); and strategies that combine CNN with recurrent models [39,40], or ConvLSTM [41,42]. Recently, attention-based architectures have been proposed for the SITS classification in the context of crop type mapping [22,43]. The work presented in [22] shows that attention-based mechanisms outperformed CNNs but are at par with LSTM on unprocessed data, e.g., cloudy optical SITS; however, when extensive data pre-processing is applied, results are comparable to random forests.…”

Section: Satellite Image Time Series Classificationmentioning

confidence: 99%

Crop Type Mapping from Optical and Radar Time Series Using Attention-Based Deep Learning

2021

View full text Add to dashboard Cite

Crop maps are key inputs for crop inventory production and yield estimation and can inform the implementation of effective farm management practices. Producing these maps at detailed scales requires exhaustive field surveys that can be laborious, time-consuming, and expensive to replicate. With a growing archive of remote sensing data, there are enormous opportunities to exploit dense satellite image time series (SITS), temporal sequences of images over the same area. Generally, crop type mapping relies on single-sensor inputs and is solved with the help of traditional learning algorithms such as random forests or support vector machines. Nowadays, deep learning techniques have brought significant improvements by leveraging information in both spatial and temporal dimensions, which are relevant in crop studies. The concurrent availability of Sentinel-1 (synthetic aperture radar) and Sentinel-2 (optical) data offers a great opportunity to utilize them jointly; however, optimizing their synergy has been understudied with deep learning techniques. In this work, we analyze and compare three fusion strategies (input, layer, and decision levels) to identify the best strategy that optimizes optical-radar classification performance. They are applied to a recent architecture, notably, the pixel-set encoder–temporal attention encoder (PSE-TAE) developed specifically for object-based classification of SITS and based on self-attention mechanisms. Experiments are carried out in Brittany, in the northwest of France, with Sentinel-1 and Sentinel-2 time series. Input and layer-level fusion competitively achieved the best overall F-score surpassing decision-level fusion by 2%. On a per-class basis, decision-level fusion increased the accuracy of dominant classes, whereas layer-level fusion improves up to 13% for minority classes. Against single-sensor baseline, multi-sensor fusion strategies identified crop types more accurately: for example, input-level outperformed Sentinel-2 and Sentinel-1 by 3% and 9% in F-score, respectively. We have also conducted experiments that showed the importance of fusion for early time series classification and under high cloud cover condition.

show abstract

Section: Satellite Image Time Series Classificationmentioning

confidence: 99%

Crop Type Mapping from Optical and Radar Time Series Using Attention-Based Deep Learning

2021

View full text Add to dashboard Cite

show abstract

“…Temporal relations between observed values in a time series are taken into account. Time series classification models for satellite data include 1D convolution neural networks (1D-CNN) [8,18], recurrent neural networks (RNN) [45], and attentionbased deep learning [46,47]. The sits package supports a set of 1D-CNN algorithms: TempCNN [8], ResNet [48], and InceptionTime [18].…”

Section: Training Machine Learning Modelsmentioning

confidence: 99%

Satellite Image Time Series Analysis for Big Earth Observation Data

et al. 2021

View full text Add to dashboard Cite

The development of analytical software for big Earth observation data faces several challenges. Designers need to balance between conflicting factors. Solutions that are efficient for specific hardware architectures can not be used in other environments. Packages that work on generic hardware and open standards will not have the same performance as dedicated solutions. Software that assumes that its users are computer programmers are flexible but may be difficult to learn for a wide audience. This paper describes sits, an open-source R package for satellite image time series analysis using machine learning. To allow experts to use satellite imagery to the fullest extent, sits adopts a time-first, space-later approach. It supports the complete cycle of data analysis for land classification. Its API provides a simple but powerful set of functions. The software works in different cloud computing environments. Satellite image time series are input to machine learning classifiers, and the results are post-processed using spatial smoothing. Since machine learning methods need accurate training data, sits includes methods for quality assessment of training samples. The software also provides methods for validation and accuracy measurement. The package thus comprises a production environment for big EO data analysis. We show that this approach produces high accuracy for land use and land cover maps through a case study in the Cerrado biome, one of the world’s fast moving agricultural frontiers for the year 2018.

show abstract

“…Our RF-based solutions are tested against several deep-learning approaches, using our own available implementations and prediction at the parcel level on T31FM tile: Recurrent Neural Networks (GRU, LSTM (Ienco et al, 2017), ConvLSTM (Rußwurm and Körner, 2018), for Gated Recurrent Unit, Long Short-Term Memory, and Convultional LSTM, respectively) and a hybrid Spatio-Temporal attention-based architecture (PSE-TAE, for Pixel-Set Encoder Temporal Attention Encoder) exhibiting state of the art results (Sainte Fare Garnot et al, 2020). We selected such methods since the temporal dimension is superior to the spatial one for Sentinel-based crop mapping (Sainte Fare Garnot et al, 2019) and since recurrent and attention-based mechanisms are superio to convolutional approaches (Rußwurm and Körner, 2020).…”

Section: Performance With Respect To Existing Solutionsmentioning

confidence: 99%

“…The vast literature dealing with crop mapping has demonstrated the relevance of satellite image time series (STIS) both at the pixel and parcel levels (Belgiu and Csillik, 2018;Sitokonstantinou et al, 2018), especially with the joint exploitation of Synthetic Aperture Radar (SAR) and optical images (Veloso et al, 2017;Neetu and Ray, 2020). In particular, deep learning techniques have recently shown their suitability to foster temporal information extraction from multi-modal STIS (Zhao et al, 2020;Adrian, Sagan, and Maimaitijiang, 2021) and their high discrimination power for a large range of crop types (Kussul et al, 2017;Ji et al, 2018;Sainte Fare Garnot et al, 2020;Rußwurm and Körner, 2020). However, they have not yet proved to be country-wise compliant.…”

Section: Introductionmentioning

confidence: 99%

Investigating operational country-level crop monitoring with Sentinel~1 and~2 imagery

Nugent

Giordano

Mallet

2021

Remote Sensing Letters

View full text Add to dashboard Cite

In this paper, we propose an operational solution for the yearly classification of crop parcels at national scale (namely France) for Land Parcel Identification System updating, under the Common Agricultural Policy (CAP) umbrella. Our pipeline is based on the ι 2 open source framework and fed with both time series of Sentinel-1 radar and Sentinel-2 optical images, with complementary contributions. Three conceivable scenarios are investigated with two sets of nomenclatures (17 and 43 classes): early, on-line, and late classifications. Experiments performed on 2017 show very satisfactory results (82-97%), locally almost on-par with state-of-the-art deep based methods. We can conclude our framework offers a strong basis for countryscale operational deployment for 2020+ CAP.

show abstract

Satellite Image Time Series Classification With Pixel-Set Encoders and Temporal Self-Attention

Cited by 78 publications

References 26 publications

Crop Type Mapping from Optical and Radar Time Series Using Attention-Based Deep Learning

Crop Type Mapping from Optical and Radar Time Series Using Attention-Based Deep Learning

Satellite Image Time Series Analysis for Big Earth Observation Data

Investigating operational country-level crop monitoring with Sentinel~1 and~2 imagery

Contact Info

Product

Resources

About