Learning features from georeferenced seafloor imagery with location guided autoencoders

Abstract

Although modern machine learning has the potential to greatly speed up the interpretation of imagery, the varied nature of the seabed and limited availability of expert annotations form barriers to its widespread use in seafloor mapping applications. This motivates research into unsupervised methods that function without large databases of human annotations. This paper develops an unsupervised feature learning method for georeferenced seafloor visual imagery that considers patterns both within the footprint of a single image frame and broader scale spatial characteristics. Features within images are learnt using an autoencoder developed based on the AlexNet deep convolutional neural network. Features larger than each image frame are learnt using a novel loss function that regularises autoencoder training using the Kullback–Leibler divergence function to loosely assume that images captured within a close distance of each other look more similar than those that are far away. The method is used to semantically interpret images taken by an autonomous underwater vehicle at the Southern Hydrates Ridge, an active gas hydrate field and site of a seafloor cabled observatory at a depth of 780 m. The method's performance when applied to clustering and content‐based image retrieval is assessed against a ground truth consisting of more than 18,000 human annotations. The study shows that the location based loss function increases the rate of information retrieval by a factor of two for seafloor mapping applications. The effects of physics‐based colour correction and image rescaling are also investigated, showing that the improved consistency of spatial information achieved by rescaling is beneficial for recognising artificial objects such as cables and infrastructures, but is less effective for natural objects that have greater dimensional variability.

Leveraging Metadata in Representation Learning With Georeferenced Seafloor Imagery

“…In our implementation, AlexNet [7] and its inverted architecture are used as the encoder and decoder, respectively, where any type of neural network can be used to construct autoencoder in a similar way. Our previous LGA method [19] can be regarded as a specific case of eq. ( 1), where only L loc and λ loc are used.…”

“…To investigate the effectiveness of the proposed regularisation, the autoencoder is trained (i) without regularisation, (ii) with L loc , (iii) with L dep , (iv) with both L loc and L dep on all 32,097 images in the dataset. AlexNet [7] with batch normalisation is used as the encoder architecture, and its inverse is used as the decoder where the number of dimensions of the encoder output (equal to the number of dimensions of the decoder input) is set to 16 in accordance with our previous work [19]. The autoencoder weights are initialised with the values of AlexNet pre-trained on ImageNet.…”

“…First, the use of metadata is advanced by incorporating depth information in parallel to horizontal location information for learning regularisation. This is significant as even though our previous LGA method used horizontal geo-location to regularise learning [19], this method cannot regularise learning across large horizontal spatial discontinuities in observation, as is often seen between different AUV dives. Additionally, although the effectiveness of the method when applied to dense survey trajectories that fully cover a 2D region of the seafloor has been demonstrated, it is not clear how effective the method is for sparse trajectories.…”

“…For image processing applications, Tile2Vec [18] extends the assumption to spatially distributed data, demonstrating its effectiveness for satellite image interpretation. In [19], we developed a Location Guided Autoencoder (LGA) that regularises autoencoder learning using horizontal geo-location information for efficient clustering and content-based retrieval of seafloor imagery. In [20], a similar assumption is introduced for CNN-based coral detection from seafloor imagery, where object tracking results in sequential frames are used for semi-supervised training.…”

Unsupervised Knowledge Transfer for Object Detection in Marine Environmental Monitoring and Exploration

“…Walker et al [31] use physics based color correction and scale normalization on underwater images to reduce the generalization error of a DeepLabV3+ model [32] for image segmentation. Similarly, Yamada et al [33] use color correction and image rescaling to enhance their method for unsupervised feature learning of georeferenced sea floor images. All methods are applied to a single dataset and are not used for knowledge transfer to enable cross-dataset machine learning.…”