Human‐in‐the‐Loop Segmentation of Earth Surface Imagery

Abstract

Segmentation, or the classification of pixels (grid cells) in imagery, is ubiquitously applied in the natural sciences. Manual methods are often prohibitively time-consuming, especially those images consisting of small objects and/or significant spatial heterogeneity of colors or textures. Labeling complicated regions of transition that in Earth surface imagery are represented by collections of mixed-pixels, -textures, and -spectral signatures, can be especially error-prone because it is difficult to reliably unmix, identify and delineate consistently. However, the success of supervised machine learning (ML) approaches is entirely dependent on good label data. We describe a fast, semi-automated, method for interactive segmentation of N-dimensional (x, y, N) images into two-dimensional (x, y) label images. It uses human-in-the-loop ML to achieve consensus between the labeler and a model in an iterative workflow. The technique is reproducible; the sequence of decisions made by human labeler and ML algorithms can be encoded to file, so the entire process can be played back and new outputs generated with alternative decisions and/or algorithms. We illustrate the scientific potential of segmentation of imagery of diverse settings and image types using six case studies from river, estuarine, and open coast environments. These photographic and non-photographic imagery consist of 1-and 3-bands on regular and irregular grids ranging from centimeters to tens of meters. We demonstrate high levels of agreement in label images generated by several labelers on the same imagery, and make suggestions to achieve consensus and measure uncertainty, ideal for widespread application in training supervised ML for image segmentation.Plain Language Summary Labeling pixels in scientific images by hand is time-consuming and error-prone, so we would like to train computers to do that for us. We can use automated techniques from Artificial Intelligence or AI, like one called Deep Learning, but it needs a lot of example images and corresponding labels that have been made by hand. So, we still need to label quite a lot of images at the pixel level-called image segmentation. We made a computer program called Doodler that speeds up the process; you label some pixels, and it labels the rest. It is the fastest method we know of for image segmentation because it is semi-automated. We also show that it produces accurate and precise labeling, as we demonstrated by having multiple people use this method to label the same images. Because it is so fast and accurate, it allows us to get enough data to train Deep Learning models to do segmentation on all the images we have, from the past and in the future. Doodler therefore enables geoscientists to use Artificial Intelligence to extract much more information from their imagery, in service of geoscience in general.

A 1.2 Billion Pixel Human-Labeled Dataset for Data-Driven Classification of Coastal Environments

“…It's use in the Coast Train project is designed in such a way that each label image may be reconstructed using the sparse annotations provided by a human annotator, and further, those annotations might be repurposed using a different algorithm, if necessary. This idea ensures reproducibility and is articulated further in a companion paper 49 that is based on a similar dataset 50 that complements the one described here but is much smaller and spatially and temporally less extensive. The level of reported detail surrounding new human-labeled datasets is often poor, including the minutiae of decisions and other details that might impact the subsequent use of the data 34 , so below we describe how the labeling team interacted over the tasks.…”

“…Mixed sand-gravel beaches are represented in our dataset using 5-cm orthomosaic imagery created from low altitude imagery collected between 2016 and 2018 at Town Neck Beach in Sandwich, Massachusetts 47 . All 5-cm orthomosaic imagery was downloaded in GeoTiff format, tiled into smaller pieces of either 1024x1024x3 or 2048x2048x3 pixels depending on the dataset using the Geospatial Data Abstraction Library software 48 and converted to jpeg format prior to use with Doodler, our labeling tool that creates dense (i.e., per pixel) labels from sparse annotations [49][50] , further described in the 'Image Labeling' section below. All imagery data are provided in unsigned eight-bit format.…”

“…Image Labeling. We achieved a fully reproducible workflow by using a semi-interactive ML program called 'Doodler' [49][50] that uses sparse annotations contributed by human labelers to estimate classes for all pixels. It's use in the Coast Train project is designed in such a way that each label image may be reconstructed using the sparse annotations provided by a human annotator, and further, those annotations might be repurposed using a different algorithm, if necessary.…”

“…It is common to divide the work of labeling data among a group of people, which allows the labeling to be carried out in a shorter time period. However, group labeling in this way does not allow for quality assurance such as flagging outliers and measuring inter-labeler agreement 49 . We adopted a hybrid approach where some datasets were labeled by a single individual, for time efficiency, but also several datasets contained many images that were labeled by two or more individuals, ensuring sets of labels that could be compared quantitatively 49 , a procedure that is described below.…”

A Reproducible and Reusable Pipeline for Segmentation of Geoscientific Imagery

“…

Deep-Learning-based image segmentation (or "semantic segmentation") starts with a research question and relevant labeled training data, that is, pairs of images and corresponding labels. Training data can come from existing sources (e.g., Wernette et al, 2022) or made from scratch using labeling tools (e.g., Buscombe et al, 2021). With training data in hand, researchers are left to wrangle, preprocess and format data, followed by building, training, and evaluating models using one of several Deep Learning frameworks.

…”