CNN, RNN, or ViT? An Evaluation of Different Deep Learning Architectures for Spatio-Temporal Representation of Sentinel Time Series

Abstract

Rich information in multi-temporal satellite images can facilitate pixel-level land cover classification. However, what is the most suitable deep learning architecture for high-dimension spatio-temporal representation of remote sensing time-series remains unclear. In this study, we theoretically analyzed the different mechanisms of the different deep learning structures, including the commonly used convolutional neural network (CNN), the high-dimension CNN (3D CNN), the recurrent neural network (RNN), and the newest vision transformer (ViT), with regard to learning and representing the temporal information for spatiotemporal data. The performance of the different models was comprehensively evaluated on large-scale Sentinel-1 and Sentinel-2 time-series images covering the whole of Slovenia. Several observations can be made. Firstly, the 3D CNN, long short-term memory (LSTM), and ViT, which all have specific structures that preserve temporal information, can effectively extract the spatiotemporal information, with the 3D CNN and ViT showing the best performance.Secondly, the performance of the 2D CNN, in which the temporal information is collapsed, is lower than that of the 3D CNN, LSTM, and ViT; however, it significantly outperforms the conventional methods of random forest (RF) and XGBoost. We also observed that using both optical and synthetic aperture radar (SAR) images performs almost the same as using only optical images, indicating that the information that can be extracted from optical images is sufficient for land-cover classification. However, when optical imaging is affected by poor weather, SAR images, as a beneficial supplement, can provide satisfactorily classification results. Finally, the modern deep learning methods can effectively overcome the disadvantages in imaging conditions where parts of an image or images of some periods are missing. The testing data are available at gpcv.whu.edu.cn/data.

Land Use and Land Cover Classification Meets Deep Learning: A Review

“…The most important feature of RNN is the hidden state throughout the input and output of the network. It is used as an input to the RNN along with the input vectors, which are updated and then used as an output to the RNN along with the output vectors [26,27]. At this point, the updated hidden state that is output will be used as part of the next input, thus preserving the previous information.…”

“…Sensors 2023, 23, 8966 4 of 27 output to the RNN along with the output vectors [26,27]. At this point, the updated hidden state that is output will be used as part of the next input, thus preserving the previous information.…”

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

“…Meanwhile, active-based views are the quintessential views chosen to complement (and improve by fusion) the optical in classification tasks with MV models, e.g. by using SAR view [1], [15], [33], [34], [36], [99], [113], [120], [124], [126], [143], [163], [173], [174], [188], [190], [193], [199]- [202] or LiDAR view [94], [139]. Furthermore, the DSM view has been widely used together with the optical view [17]- [19], [55], [58], [64], [65], [76], [80], [84], [85], [88], [89], [92], [104], [123], [184], where on some occasions is a LiDAR-derived DSM [32], [34], [35], [59], [62], [81], [86], [87], [93], …”

“…These works show that the predictive performance improves with respect to training on any of the single-views, e.g. with optical and activebased views (SAR/LiDAR/DSM) [1], [13], [13], [15], [18], [32]- [35], [58], [59], [65], [81], [86], [92]- [94], [96], [99], [102], [113], [133], [136], [143], [163], [169], [190], [191], [193], [199], [202]. This indicates that the views complement each other in the MV learning for EO tasks, in addition to the fact that there is evidence when other diverse views are chosen to supplement or replace the optical view [16], [70], [71], [106], [117], [159], [160], [181], [183], [210], [211].…”

Agricultural Land Cover Mapping through Two Deep Learning Models in the Framework of EU’s CAP Activities Using Sentinel-2 Multitemporal Imagery

“…Prominent examples are satellite image fusion for improved land use/land cover classification [35], object detection [36,37], and change detection [38,39] in remote sensing images, as well as the delineation of agricultural fields from satellite images [40]. CNNs are appealing to the remote sensing community due to their inherent nature to exploit the two-dimensional structure of images, efficiently extracting spectral and spatial features, while RNNs can handle sequential input in continuous dimensions with sequential long-range dependency, thus making them appropriate for the analysis of the spectral-temporal information in time series stacks [19,34,[41][42][43][44].…”