Crop mapping from image time series: deep learning with multi-scale label hierarchies

Abstract

The aim of this paper is to map agricultural crops by classifying satellite image time series. Domain experts in agriculture work with crop type labels that are organised in a hierarchical tree structure, where coarse classes (like orchards) are subdivided into finer ones (like apples, pears, vines, etc.). We develop a crop classification method that exploits this expert knowledge and significantly improves the mapping of rare crop types. The three-level label hierarchy is encoded in a convolutional, recurrent neural network (con-vRNN), such that for each pixel the model predicts three labels at different level of granularity. This end-to-end trainable, hierarchical network architecture allows the model to learn joint feature representations of rare classes (e.g., apples, pears) at a coarser level (e.g., orchard ), thereby boosting classification performance at the fine-grained level. Additionally, labelling at different granularity also makes it possible to adjust the output according to the classification scores; as coarser labels with high confidence are sometimes more useful for agricultural practice than fine-grained but very uncertain labels. We validate the proposed method on a new, large dataset that we make public. ZueriCrop covers an area of 50 km × 48 km in the Swiss cantons of Zurich and Thurgau with a total of 116'000 individual fields spanning 48 crop classes, and 28,000 (multi-temporal) image patches from Sentinel-2. We compare our proposed hierarchical convRNN model with several baselines, including methods designed for imbalanced class distributions. The hierarchical approach performs superior by at least 9.9 percentage points in F1-score.

AgriSen-COG, a Multicountry, Multitemporal Large-Scale Sentinel-2 Benchmark Dataset for Crop Mapping Using Deep Learning

“…ZueriCrop [11] is a dataset for crop type mapping based on Sentinel-2 Level-2A bottomof-atmosphere images from the Switzerland area (50 km × 48 km area). The GT data were extracted from Switzerland's LPIS (Swiss Federal Office for Agriculture), which is not publicly available.…”

Crop Classification Under Varying Cloud Cover With Neural Ordinary Differential Equations

“…The ZueriCrop dataset was generated by [53]. This dataset is collected a time series of 71 multi-spectral Sentinel-2 satellite images with a ground resolution of 10 m collected over a 50 × 48 km 2 area over the Swiss cantons of Zurich and Thurgau (Fig.…”

Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs

“…The marginalisation loss summarizes the hierarchical information in a bottom-up manner and, although being one of the simplest approaches, emerged as one of the most effective. In [15] the authors investigate the task of classifying agricultural crops from a sequence of satellite images, where the crop labels also exhibit a hierarchical structure (e.g., wheat is more similar to other cereals than to, say, orchards). They propose a convolutional recurrent architecture, where increasing depth in the spatial/convolutional dimension corresponds to a finer hierarchy level, thus deriving higher-level features for finer classification from coarser lower-level features.…”

“…Moreover, we include the taxonomic hierarchy to improve model performance at inference time. Hierarchically structured class labels can be beneficial in two different ways: on the one hand, the hierarchy can be used as a regularisation of the model, which has been shown to improve the classification of rare classes [15]; on the other hand, the hierarchy can also be used at inference time to provide a prediction (at a coarser level) for species not present in the list of the output classes. We investigate different strategies to exploit the side information, and empirically compare them.…”