When an unmanned surface vehicle (USV) navigates in narrow waterway scenarios, its ability to detect vanishing points accurately and quickly is highly important for safeguarding its navigation safety and realizing automated navigation. We propose a novel approach for detecting vanishing points based on an improved lightweight AlexNet. First, a similarity evaluation calculation method based on image texture features is proposed, by which some scenarios are selected from the filtered Google Street Road Dataset (GSRD). These filtered scenarios, together with the USV Inland Dataset (USVID), compose the training dataset, which is manually labeled according to a non-uniformly distributed grid level. Next, the classical AlexNet was adjusted and optimized by constructing sequential connections of four convolutional layers and four pooling layers and incorporating the Inception A and Inception C structures in the first two convolutional layers. During model training, we formulate vanishing point detection as a classification problem using an output layer with 225 discrete possible vanishing point locations. Finally, we compare and analyze the labeled vanishing point with the detected vanishing point. The experimental results show that the accuracy of our method and the state-of-the-art algorithmic vanishing point detector improves, indicating that our improved lightweight AlexNet can be applied in narrow waterway navigation scenarios and can provide a technical reference for autonomous navigation of USVs.