In order to complete automatic route planning in the complex navigation environment, this paper proposes a quadratic optimization genetic algorithm combining the motion characteristics of the ships. Furthermore, the constraints of the maneuvering characteristics of ship are considered to calculate the accurate planning routes fast. First of all, the turning and speed reduction model of ships are established, which is the foundation for the accurate calculation of approach states between the own ship and target ships. Second, in order to check the position relationship between the planned route and the land objects (such as the islands and rocks) on the chart, a detection algorithm between lines and irregular polygon boundary based on quadtree method is acquired. Third, a new genetic algorithm based on double-cycling optimization is proposed by this paper. Aiming at the complex problem of comprehensive collision avoidance planning under the conditions of static and dynamic obstacles, this paper realizes an efficient and feasible solution of dynamic programming for ship route. Finally, the algorithm is verified on a test fireboat, and the experiment data prove the feasibility and effectiveness of the proposed algorithm by this paper.