A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision

Abstract

Spatiotemporal information of the vehicles on a bridge is important evidence for reflecting the stress state and traffic density of the bridge. A methodology for obtaining the information is proposed based on computer vision technology, which contains the detection by Faster region‐based convolutional neural network (Faster R‐CNN), multiple object tracking, and image calibration. For minimizing the detection time, the ZF (Zeiler & Fergus) model with five convolutional layers is selected as the shared part between Region Proposal Network and Fast R‐CNN in Faster R‐CNN. An image data set including 1,694 images is established about eight types of vehicles for training Faster R‐CNN. Combined with the detection of each frame of the video, the methods of multiple object tracking and image calibration are developed for acquiring the vehicle parameters, including the length, number of axles, speed, and the lane that the vehicle is in. The method of tracking is mainly based on the judgment of the distances between the vehicle bounding boxes in virtual detection region. As for image calibration, it is based on the moving standard vehicles whose lengths are known, which can be regarded as the 3D template to calculate the vehicle parameters. After acquiring the vehicles' parameters, the spatiotemporal information of the vehicles can be obtained. The proposed system has a frame rate of 16 fps and only needs two cameras as the input device. The system is successfully applied on a double tower cable‐stayed bridge, and the identification accuracies of the types and number of axles are about 90 and 73% in the virtual detection region, and the speed errors of most vehicles are less than 6%.

Deep learning for data anomaly detection and data compression of a long‐span suspension bridge

“…Deep learning has also been used for vehicle spatial and temporal distribution monitoring in bridges. Zhang, Zhou, and Zhang (). proposed a Faster RCNN‐based method that can obtain the vehicle information automatically in real time.…”

A review of computer vision–based structural health monitoring at local and global levels

“…This part discusses about the challenges and opportunities in CV-SHM. Chen et al 219 : long-span bridge (load spectrum) Background subtracking + ZNCC template matching Chen et al 220 : long-span cable-stayed bridge (load spatio-temporal distribution) HOG feature + Random Forest + non-maximum suppression Pan et al 221 : road barrier (vehicle distribution and speed) Faster R-CNN Zhang et al 222 : long-span cable-stayed bridge Kalman filter tracking + WIM Dan et al 223 : highway bridge (load and spatio-temporal distribution) Adaptive thresholding + morphological reconstruction + WIM Micu et al 224 : long-span suspension bridge (vehicle load) CNN for classification + Faster R-CNN + Kalman filter tracking Zhou et al 225 : highway bridge (vehicle load range, spatio-temporal distribution, and tracking) CV: computer vision; DIC: digital image correlation; PIV: particle image velocimetry; R-CNN: region-based convolutional neural network; UIL: unit influence line; UIS: unit influence surface; ZNCC: zero-mean normalized cross-correlation; HOG: histogram of gradient; WIM: weigh-in-motion.…”

Deep Learning Based Vehicle Detection and Classification Methodology Using Strain Sensors under Bridge Deck

“…Due to the computer vision technology, the image sensor has been applied in vehicle detection. Videos are obtained from the perspective of front or side view [ 17 , 18 ]. Image processing techniques and machine learning methods were used to detect the vehicles in continuous frames.…”

“…By applying CNN-based object detection methods, the vehicles were segmented from the background with class labels [ 27 , 28 , 29 ]. When calibrated with real world coordination systems, the spacing information of vehicles was accessed [ 18 ]. Combining the image processing-based clustering method, the axles were detected and thus the axle type was obtained [ 30 ].…”