Multi-Task Classification of Sewer Pipe Defects and Properties using a Cross-Task Graph Neural Network Decoder

Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

“…On the F2 CIW metric we observe an increase between 0.7 and 2.5 percentage points, with the largest increase observed on the ResNet-50, where the performance is improved by 2.4-2.5 percentage points on both the validation and testing splits. This is significantly better than the benchmark algorithm from Haurum and Moeslund [16], and a comparable performance to the previous best performing model on Sewer-ML, the multi-task classification method CT-GAT [37], while only using the sewer defect labels during training. This demonstrates that it is possible to significantly increase the sewer defect classification performance without needing auxiliary data such as water level, pipe shape, and pipe material.…”

“…The field has, however, become more transparent as many have started to directly compare different methods on the same datasets, in an effort to offset the lack of public detection and segmentation datasets [17,36,34]. Recently, the field has also started investigating other parts of the sewer inspection process [30,32,17,[37][38][39][40][41], such as Haurum et al [37] proposing a multi-task classification approach for simultaneously classifying defects, water level, pipe material, and pipe shape, and Wang et al [30] proposed a framework to accurately determine the severity of defects related to the operation and maintenance of the pipes. The field has also adopted recent trends from the general computer vision field such as selfsupervised learning [39], synthetic data generation [25,24,[42][43][44], neural architecture search [45], and usage of the Transformer architecture [17,46], indicating that the automated sewer inspection field is catching up to the general computer vision domain.…”

“…We follow the training procedure of Haurum et al [37] with the addition of using the Exponential Moving Average (EMA) technique on the model weights, see Table 1. We utilize the Fourier Network (FNet) based attention mechanism [73] in the HViT as an efficient alternative to the conventional MHSA based attention mechanism.…”

“…We compare using both the conventional patch based tokenizer and the proposed Sinkhorn tokenizer. Lastly, we compare to the previously published results on Sewer-ML [16,37]. We run all experiments within the same codebase, using the torchvision [78], Pytorch Lightning [79] and timm [80] libraries.…”

“…Best performance per column is denoted in bold. We also include the previous published results on Sewer-ML [16,37], and HViT-like models [7,13,14]. *denotes that the method was trained in a multi-task classification framework.…”