Towards operational phytoplankton recognition with automated high-throughput imaging, near-real-time data processing, and convolutional neural networks

Abstract

Plankton communities form the basis of aquatic ecosystems and elucidating their role in increasingly important environmental issues is a persistent research question. Recent technological advances in automated microscopic imaging, together with cloud platforms for high-performance computing, have created possibilities for collecting and processing detailed high-frequency data on planktonic communities, opening new horizons for testing core hypotheses in aquatic ecosystems. Analyzing continuous streams of big data calls for development and deployment of novel computer vision and machine learning systems. The implementation of these analysis systems is not always straightforward with regards to operationality, and issues regarding data flows, computing and data treatment need to be considered. We created a data pipeline for automated near-real-time classification of phytoplankton during remote deployment of imaging flow cytometer (Imaging FlowCytobot, IFCB). Convolutional neural network (CNN) is used to classify continuous imaging data with probability thresholds used to filter out images not belonging to our existing classes. The automated data flow and classification system were used to monitor dominating species of filamentous cyanobacteria on the coast of Finland during summer 2021. We demonstrate that good phytoplankton recognition can be achieved with transfer learning utilizing a relatively shallow, publicly available, pre-trained CNN model and fine-tuning it with community-specific phytoplankton images (overall F1-score of 0.95 for test set of our labeled image data complemented with a 50% unclassifiable image portion). This enables both fast training and low computing resource requirements for model deployment making it easy to modify and applicable in wide range of situations. The system performed well when used to classify a natural phytoplankton community over different seasons (overall F1-score 0.82 for our evaluation data set). Furthermore, we address the key challenges of image classification for varying planktonic communities and analyze the practical implications of confused classes. We published our labeled image data set of Baltic Sea phytoplankton community for the training of image recognition models (~63000 images in 50 classes) to accelerate implementation of imaging systems for other brackish and freshwater communities. Our evaluation data set, 59 fully annotated samples of natural communities throughout an annual cycle, is also available for model testing purposes (~150000 images).

Robust detection of marine life with label-free image feature learning and probability calibration

“…While high accuracy can be obtained on 'rebalanced' test data with equal representation of all classes (Branco et al 2015, Kraft et al 2022, this is not representative of real-world performance. (Bochinski et al 2019) handle an imbalanced dataset by first training on a balanced subdivision of the complete dataset, only sampling as many images for each class as are available for the least present class.…”

“…A number of plankton classification studies have used probability filtering, in which automatic classification results are discarded when the maximum inferred probability is below some (possibly class-specific) threshold (Luo et al 2018, Guo et al 2021, Kraft et al 2022. This technique reduces the number of incorrect images, but leaves some images unclassified, possibly biasing the estimated class frequencies or relationships with environmental drivers.…”

Towards Phytoplankton Parasite Detection Using Autoencoders

“…The phytoplankton data from Utö IFCB can be currently classified near real-time into 50 different classes as described by [48]. Putative parasite infection images were manually annotated by experts using another Utö data collected between February-August 2021, using phytoplankton data from nine classes.…”

“…In our experiments, we use a phytoplankton anomaly dataset derived from the annotated images used to train the classifier described in the previous paragraph with the OK samples from the dataset published in [48] and the NOK samples from the unpublished 2021 Utö data. It contains over 6200 manually annotated and expert-validated samples throughout 9 plankton species with known anomalies as is shown in Table 1.…”

RMP-Net: A structural reparameterization and subpixel super-resolution-based marine scene segmentation network

“…In the process of using autonomous underwater explorers to explore the seabed, the detection and perception accuracy of the vision system is relatively high. For example, for challenging problems such as insufficient ambient light, related research studies have used convolutional neural networks to detect and identify marine organisms (Li et al, 2020;Xu et al, 2021;Zhang et al, 2021;Kraft et al, 2022). However, using target detection to achieve target object positioning is still not accurate enough.…”