Leak monitoring is essential for the intelligent operation and maintenance of marine systems, and can effectively prevent catastrophic accidents on ships. In response to this challenge, a machine vision-based leak model is proposed in this study and applied to leak detection in different types of marine system in complex engine room environments. Firstly, an image-based leak database is established, and image enhancement and expansion methods are applied to the images. Then, Standard Convolution and Fast Spatial Pyramid Pooling modules are added to the YOLOv5 backbone network to reduce the floating-point operations involved in the leak feature channel fusion process, thereby improving the detection speed. Additionally, Bottleneck Transformer and Shuffle Attention modules are introduced to the backbone and neck networks, respectively, to enhance the feature representation performance, select critical information for the leak detection task, and suppress non-critical information to improve detection accuracy. Finally, the proposed model’s effectiveness is verified using leak images collected by the ship’s video system. The test results demonstrate that the proposed model exhibits excellent recognition performance for various types of leak, especially for drop-type leaks (for which the accuracy reaches 0.97).