The new way of offshore operation and maintenance based on unmanned ships has outstanding advantages. Aiming at the problem of lack of overall understanding of the complex environment above and under the water surface during the operation and maintenance of unmanned ships, a stitching and fusion technology of overwater and underwater visual images for unmanned ships is proposed. The software and hardware framework of the overwater and underwater visual image fusion system is constructed, the image processing methods in different environments are defined, and the accurate acquisition of obstacle information is realized. In the two experimental scenarios, the stitching accuracy of the obstacle model based on an extended neighborhood method can reach more than 85% within the obstacle distance of 8m and more than 80% within the obstacle distance of 14m. An image-driven Frustum-PointNets detection algorithm is proposed to obtain comprehensive obstacle avoidance information. In addition, the average accuracy of the three-dimensional detection of the algorithm is up to 91.40%. These results are significant and have a good reference value, as it demonstrates that the stitching and fusion method can effectively obtain the comprehensive information of overwater and underwater objects for unmanned ship.

Aiming at the problem that multi-ship target detection and tracking based on cameras is difficult to meet the accuracy and speed requirements at the same time in some complex scenes, an improved YOLOv4 algorithm is proposed, which simplified the network of the feature extraction layer to obtain more shallow feature information and avoid the disappearance of small ship target features, and uses the residual network to replace the continuous convolution operation to solve the problems of network degradation and gradient disappearance. In addition, a nonlinear target tracking model based on the UKF method is constructed to solve the problem of low real-time performance and low precision in multi-ship target tracking. Multi-ship target detection and tracking experiments were carried out in many scenes with large differences in ship sizes, strong background interference, tilted images, backlight, insufficient illumination, and rain. Experimental results show that the average precision of the detection algorithm of this paper is 0.945, and the processing speed is about 34.5 frame per second, where the real-time performance is much better than other algorithms while maintaining high precision. Furthermore, the multiple object tracking accuracy (MOTA) and the multiple object tracking precision (MOTP) of this paper algorithm are 76.4 and 80.6, respectively, which are both better than other algorithms. The method proposed in this paper can realize the ship target detection and tracking well, with less missing detection and false detection, and also has good accuracy and real-time performance. The experimental results provide a valuable theoretical reference for the further practical application of the method.