The unmanned aerial vehicle (UAV) is inexpensive and offers a fast response speed and robust flexibility; thus, it is a promising tool in the maritime buoyage inspection scenario, which involves monitoring and accessing a lateral mark system far away from the coast. However, two main problems can occur during inspection. The first is extreme weather conditions, resulting in a deviation between the inspection route and the design route. The second is that the buoyage beacons are visited only once. Therefore, this article proposes a buoyage inspection system consisting of a single UAV and random coastal buoys. The UAV automatically takes off from the depot, performs a self-check on the buoy beacons, and then returns to the depot. A cascade active disturbance rejection controller (ADRC) is designed to adjust the real-time trajectory of the UAV system. A feasible trajectory planning method is also designed based on the continuous Hopfield neural network (CHNN) and genetic algorithm (GA) to minimize the inspection distance. Extensive simulations are conducted to demonstrate the effectiveness of the proposed method.