Assessing Representation Learning and Clustering Algorithms for Computer-Assisted Image Annotation—Simulating and Benchmarking MorphoCluster

Abstract

Image annotation is a time-consuming and costly task. Previously, we published MorphoCluster as a novel image annotation tool to address problems of conventional, classifier-based image annotation approaches: their limited efficiency, training set bias and lack of novelty detection. MorphoCluster uses clustering and similarity search to enable efficient, computer-assisted image annotation. In this work, we provide a deeper analysis of this approach. We simulate the actions of a MorphoCluster user to avoid extensive manual annotation runs. This simulation is used to test supervised, unsupervised and transfer representation learning approaches. Furthermore, shrunken k-means and partially labeled k-means, two new clustering algorithms that are tailored specifically for the MorphoCluster approach, are compared to the previously used HDBSCAN*. We find that labeled training data improve the image representations, that unsupervised learning beats transfer learning and that all three clustering algorithms are viable options, depending on whether completeness, efficiency or runtime is the priority. The simulation results support our earlier finding that MorphoCluster is very efficient and precise. Within the simulation, more than five objects per simulated click are being annotated with 95% precision.

A computer vision-based approach for estimating carbon fluxes from sinking particles in the ocean

“…Compared to statisticsbased classification methods (Trudnowska et al, 2021), this approach is based on a categorization scheme derived from pre-defined carbon flux pathways with known ecological significance. Furthermore, like methods developed for similar applications (Schröder et al, 2020;Schröder and Kiko, 2022), our method drastically reduces the amount of human effort required for obtaining classification with the added net benefit that all particles are assigned a label. The human-in-the-loop domain adaptation approach demonstrated here is one that could be applied not only to our marine particle dataset, but any dataset that is subject to distribution shift and a scarcity of labels for minority classes, two challenges which are ubiquitous in ecological image datasets.…”

“…CNNs have also been applied in semi-supervised approaches, which require the human annotator to review only a fraction of imaged particles while clustering similar images together (Schröder et al, 2020;Schröder and Kiko, 2022). This approach has the potential to reduce the subjectivity of a human annotator, but its success depends on how well the clustering algorithm can assign images to ecologically important categories.…”