Automated plankton image analysis using convolutional neural networks

Abstract

The rise of in situ plankton imaging systems, particularly high-volume imagers such as the In Situ Ichthyoplankton Imaging System, has increased the need for fast processing and accurate classification tools that can identify a high diversity of organisms and nonliving particles of biological origin. Previous methods for automated classification have yielded moderate results that either can resolve few groups at high accuracy or many groups at relatively low accuracy. However, with the advent of new deep learning tools such as convolutional neural networks (CNNs), the automated identification of plankton images can be vastly improved. Here, we describe an image processing procedure that includes preprocessing, segmentation, classification, and postprocessing for the accurate identification of 108 classes of plankton using spatially sparse CNNs. Following a filtering process to remove images with low classification scores, a fully random evaluation of the classification showed that average precision was 84% and recall was 40% for all groups. Reliably classifying rare biological classes was difficult, so after excluding the 12 rarest taxa, classification accuracy for the remaining biological groups became > 90%. This method provides proof of concept for the effectiveness of an automated classification scheme using deep-learning methods, which can be applied to a range of plankton or biological imaging systems, with the eventual application in a variety of ecological monitoring and fisheries management contexts.

Improving plankton image classification using context metadata

“…Since 2012, “Deep Learning” algorithms (Krizhevsky et al ; LeCun and Ranzato ; LeCun et al ) have outperformed feature‐based classifiers in a variety of fields, including natural language processing (Socher et al ), time series analysis (Graves et al ), variational autoencoders (algorithms that learn to generate or alter existing data, such as image correction; Kingma and Welling ), plankton image analysis (Orenstein et al ; Dai et al ; Dieleman et al ; Graff and Ellen ; Wang et al ; Zheng et al ; Orenstein and Beijbom ; Luo et al ), et al Multiple algorithms have been characterized as examples of Deep Learning, the commonality being the use of repetitive layers of algorithmic structure that operate on the prior layers rather than the original input. Deep Learning algorithms tend to require orders of magnitude more computation, although often such computations are highly parallelizable and can be done rapidly given appropriate hardware.…”

“…Applications of CNN and Random Forest to phytoplankton image classification include Orenstein et al (), while further applications of CNNs to coral, plankton, and fish classification are surveyed by Moniruzzaman et al (). A detailed end‐to‐end workflow utilizing CNNs to classify large numbers of plankton images is described in Luo et al (), and the results they provide for their selected CNN architecture clearly illustrate that CNNs can be used to classify plankton images.…”

Globally Consistent Quantitative Observations of Planktonic Ecosystems

“…Additionally, to obtain taxonomic information from optical or imaging methods, there is a need for a computer-assisted human expert to classify organisms based on their optical properties (e.g., "gating" in flow cytometry) or on their image. While machine-learning methods are getting progressively more efficient (Luo et al, 2017), the final taxonomic resolution is often limited, and may include substantial errors (Culverhouse et al, 2003(Culverhouse et al, , 2006. The increased capabilities in automated recognition of images still needs to be complemented with taxonomic expertise.…”

“…The system is equipped with environmental sensors, including CTD, fluorometer, dissolved oxygen, and PAR sensors. The volume resolution of the image captures a wide taxonomic range of mesozooplankton and with lower resolution, large protists and cyanobacteria (Luo et al, 2017). The data are transferred to an onboard computer via fiber optic in real time while the platform is being towed at 2.5 m s −1 , either undulating or at fixed-depth mode to a maximum depth of 150 m.…”

Limnology and Oceanography Bulletin Volume 26 Number 4 November 2017 95-122

“…Support vector machines have also been applied to automating the recognition of objects in huge sets of marine imagery (e.g., Beijbom et al 2015). Deep machine learning methods, such as convolutional neural networks, are now also being used for this analysis (e.g., Luo et al 2017). However, statistical and machine learning methods are often like a black box, and determining which method is most appropriate is often difficult.…”