In order to realize the autonomous navigation of ships in the complex waters under the traffic separation system (TSS), taking the Chengshanjiao TSS waters as an example, an improved autonomous collision avoidance algorithm is proposed based on the artificial potential field (APF) method. By establishing five kinds of potential fields and their force functions, including the gravitational function of the goal point, the repulsion function of the target ships (TS), the gravitational function of the centerline of the traffic lane, the velocity function of the ship, and the boundary repulsion function of the navigation lane, finally the resultant force of them is calculated to guide the navigation direction of the ship, during the navigation process in this TSS water. Multiple groups of comparative experiments demonstrated that under the improved algorithm, the ship could avoid collisions with TS in four typical scenarios, according to the requirements of the 1972 International Regulations for Preventing Collisions at Sea (COLREGS) and good seamanship practice, simultaneously the ship could proceed in the navigable lane and keep safe distance with the boundaries and separation zones. The method proposed in this paper is expected to provide basic theoretical basis and technical support for the safe navigation of autonomous ships in TSS waters.