Automated plankton classification from holographic imagery with deep convolutional neural networks

Abstract

In situ digital inline holography is a technique which can be used to acquire high‐resolution imagery of plankton and examine their spatial and temporal distributions within the water column in a nonintrusive manner. However, for effective expert identification of an organism from digital holographic imagery, it is necessary to apply a computationally expensive numerical reconstruction algorithm. This lengthy process inhibits real‐time monitoring of plankton distributions. Deep learning methods, such as convolutional neural networks, applied to interference patterns of different organisms from minimally processed holograms can eliminate the need for reconstruction and accomplish real‐time computation. In this article, we integrate deep learning methods with digital inline holography to create a rapid and accurate plankton classification network for 10 classes of organisms that are commonly seen in our data sets. We describe the procedure from preprocessing to classification. Our network achieves 93.8% accuracy when applied to a manually classified testing data set. Upon further application of a probability filter to eliminate false classification, the average precision and recall are 96.8% and 95.0%, respectively. Furthermore, the network was applied to 7500 in situ holograms collected at East Sound in Washington during a vertical profile to characterize depth distribution of the local diatoms. The results are in agreement with simultaneously recorded independent chlorophyll concentration depth profiles. This lightweight network exemplifies its capability for real‐time, high‐accuracy plankton classification and it has the potential to be deployed on imaging instruments for long‐term in situ plankton monitoring.

Plankton classification with high-throughput submersible holographic microscopy and transfer learning

“…Due to its simplicity, DIHM can easily be incorporated into various cell imaging configurations including amplitude and phase images [ 21 ] and to date, numerous studies have used holography to image marine plankton [ 8 , 22 – 25 ]. There is increasing interest to use its advantages towards automating classification of plankton and particulates from water samples (e.g., [ 26 – 28 ]). A review of holographic microscopes for aquatic imaging can be found in Nayak et al [ 29 ].…”

A Review of Holography in the Aquatic Sciences: In situ Characterization of Particles, Plankton, and Small Scale Biophysical Interactions

“…For near real-time observations, data transmission is one of the main bottlenecks due to the large file sizes; rapid onboard processing to provide simplified data (e.g., organism presence and counts) in a compressed format, instead of transmitting entire images could help alleviate this problem. For this to work, is it key to develop not only fast and efficient automated classification algorithms, but also those feasible to be developed in a light-weight architecture (Guo et al, 2021). Previous and ongoing efforts toward automated classification of detected particles using various machine learning techniques, including convolutional neural networks, still need holograms to be reconstructed and processed (Davies et al, 2015;Bianco et al, 2020).…”

“…Previous and ongoing efforts toward automated classification of detected particles using various machine learning techniques, including convolutional neural networks, still need holograms to be reconstructed and processed (Davies et al, 2015;Bianco et al, 2020). Recent work has focused on the application of deep learning techniques to extract features from the interference patterns recorded on the raw holograms (Shao et al, 2020;Guo et al, 2021). Successful translation to in situ applications would help revolutionize the field, as this would enable skipping the holographic reconstruction step, greatly reducing processing times and facilitating near real-time data dissemination.…”

Computationally efficient processing of in situ underwater digital holograms

“…3. As in Davies et al (2015), Guo et al (2021), and (Nayak et al 2018), the first processing step is background subtraction. Nonuniform light levels, scratches in the camera Bottom: Raw holograms with interference from optical turbulence.…”

“…Processing schemes that avoid high-resolution reconstruction would enable analysis of larger (e.g., long time series) data sets and data from larger imaging volume DIHMs. Guo et al (2021) addresses the computational cost of hologram reconstruction by employing deep learning to classify particles based on their diffraction patterns in the raw hologram, eliminating the need for reconstruction. When a particle is relatively close to the camera, the diffraction pattern tends to resemble the shape of the target.…”