As the largest modern passenger Roll-on Roll-off (RoRo) terminal around the world, the berthing operation of Xuwen terminal is occasionally suspended due to bad weather, such as strong wind or thick fog. During the suspension, the number of stranded passengers and vehicles increasingly accumulates. As soon as the weather permits, the growth exerts great pressure, especially on large-scale vessels leaving the port, whose inefficiency may cause a loss of access to the terminal for inbound ships and chaos for port management. The focus of this study is to improve the efficiency of departure scheduling by optimizing traffic rules in the harbor basin. A mathematical optimization model is formulated for minimizing the total scheduling time, and then an adaptive simulated annealing (ASA) algorithm is proposed to solve the model. A specific decoding rule is introduced, referring to the characteristics of the mentioned model. After employing the operation data of the Xuwen terminal, a numerical experiment showed that the proposed scheduling method outperformed the first-come, first-served (FCFS) strategy and an improved ant colony algorithm (ACA). Moreover, the constructed simulation model of the terminal manifested the validity of the optimal solution.