Data science competition for cross-site delineation and classification of individual trees from airborne remote sensing data

Abstract

Delineating and classifying individual trees in remote sensing data is challenging. Many tree crown delineation methods have difficulty in closed-canopy forests and do not leverage multiple datasets. Methods to classify individual species are often accurate for common species, but perform poorly for less common species and when applied to new sites. We ran a data science competition to help identify effective methods for delineation of individual crowns and classification to determine species identity. This competition included data from multiple sites to assess the methods' ability to generalize learning across multiple sites simultaneously, and transfer learning to novel sites where the methods were not trained. Six teams, representing 4 countries and 9 individual participants, submitted predictions. Methods from a previous competition were also applied and used as the baseline to understand whether the methods are changing and improving over time. The best delineation method was based on an instance segmentation pipeline, closely followed by a Faster R-CNN pipeline, both of which outperformed the baseline method. However, the baseline (based on a growing region algorithm) still performed well as did the Faster R-CNN. All delineation methods generalized well and transferred to novel forests effectively. The best species classification method was based on a two-stage fully connected neural network, which significantly outperformed the baseline (a random forest and Gradient boosting ensemble). The classification methods generalized well, with all teams training their models using multiple sites simultaneously, but the predictions from these trained models generally failed to transfer effectively to a novel site. Classification performance was strongly influenced by the number of field-based species IDs available for training the models, with most methods predicting common species well at the training sites. Classification errors (i.e., species misidentification) were most common between similar species in the same genus and different species that occur in the same habitat. The best methods handled class imbalance well and learned unique spectral features even with limited data. Most methods performed better than baseline in detecting new (untrained) species, especially in the site with no training data. Our experience further shows that data science competitions are useful for comparing different methods through the use of a standardized dataset and set of evaluation criteria, which highlights promising approaches and common challenges, and therefore advances the ecological and remote sensing field as a whole.

Capturing long‐tailed individual tree diversity using an airborne imaging and a multi‐temporal hierarchical model

“…Scholl et al (2020) followed with a four species model at the Niwot Ridge NEON site in Colorado (NIWO). Two data science competitions, one focused on the OSBS site (Marconi et al, 2019) and the second combining data from the OSBS, Talladega, Alabama (TALL) and Mountain Lake, Virginia (MLBS) sites in the Southeastern United States (Graves et al, 2021), used NEON forest plot data with 33 species, of which only 15 had more than five individuals. The lack of data for less common species was the primary factor in poor model performance.…”

“…The lack of data for less common species was the primary factor in poor model performance. For example, as part of the multisite data competition (Graves et al, 2021), Scholl et al (2021) modeled 27 selected species classes, but only seven of these classes had non-zero evaluation accuracy. Marconi et al (2022) attempted the first NEON-wide model for 77 species across 27 sites using a pixel-based ensemble machine learning classifier.…”

“…The lack of data for less common species was the primary factor in poor model performance. For example, as part of the multisite data competition (Graves et al., 2021), Scholl et al. (2021) modeled 27 selected species classes, but only seven of these classes had non‐zero evaluation accuracy.…”

Capturing long-tailed individual tree diversity using an airborne multi-temporal hierarchical model

“…Scholl et al (2020) followed with a four species model at the Niwot Ridge NEON site in Colorado (NIWO). Two data science competitions, one focused on the OSBS site (Marconi et al 2019) and the second combining data from the OSBS, Talladega, Alabama (TALL), and Mountain Lake, Virginia (MLBS) sites in the Southeastern US (Graves et al 2021), used NEON forest plot data with 33 species, of which only 15 had more than 5 individuals. The lack of data for less common species was the primary factor in poor model performance.…”

“…The lack of data for less common species was the primary factor in poor model performance. For example, as part of the multi-site data competition (Graves et al 2021), Scholl et al (2021) modeled 27 selected species classes, but only 7 of these classes had non-zero evaluation accuracy. Marconi et al (2021) attempted the first NEON-wide model for 77 species across 27 sites using a pixel-based ensemble machine learning classifier.…”