The accurate environment potential field (EPF) modeling method and highly efficient collision avoidance (CA) approach are key technologies for maritime autonomous surface ships (MASS). A novel and accurate environment potential field (EPF) model is proposed using electronic navigation chart (ENC) face objects to describe different types of navigable and non-navigable areas, and an improved artificial potential field (APF) method is presented to realize collaborative CA and obstacle avoidance (OA). Implicit equations of complex-shaped face objects were constructed based on R-function theory, and the discrete-convex hull method was introduced to realize automatic EPF modeling. Collaborative CA and OA experiments in restricted waters were conducted on a ship handling simulator. The results show that the improved APF method can obtain a robust and deterministic collision-free path under different weather conditions and in restricted waters, and the track zone width remains within 0.1 nm. The proposed face object EPF model is efficient and accurate, even with numerous vertices and complex shapes, and can drive the ship apart at a relatively safe distance in accordance with the recommended CA parameter. We present a practical CA approach and an effective EPF modeling method for APF-based ship path planning.