This study aimed to solve the problem that is the frequent switching between the acceleration and braking modes of the driverless ferry vehicle, affecting the comfort and stability of speed control. The driverless ferry vehicle encounters unknown obstacles on the road that affect the normal planning and tracking control of the ferry vehicle and finally lead to the problem that the driverless ferry vehicle cannot drive normally. First of all, in the longitudinal control, the fuzzy PID control algorithm was utilized to produce the fuzzy PID acceleration controller by taking into account the difference between the actual and expected speeds and choosing the triangular membership function. According to the relationship between the brake oil pressure and brake torque, the brake controller was designed. The acceleration/braking switching module with acceleration tolerance zone was added to the longitudinal controller, and the acceleration/braking mode-switching controller was designed. Secondly, in the lateral control, the tire cornering stiffness was analyzed, an MPC controller with a planning module was designed, and a lateral motion controller with an obstacle avoidance replanning function was proposed. Finally, according to the prediction time domain of different planning modules corresponding to different speeds, a coordinated control strategy of horizontal and longitudinal motion was proposed by using a real-time speed adjustment planning module to predict the time domain. Through the joint simulation analysis of MATLAB and CarSim, the results show that the driving stability of the ferry vehicle was significantly improved, and the longitudinal speed error of the ferry vehicle was reduced by 43.59%. The ferry’s avoidance of obstacles and tracking of reference trajectories were significantly improved, so that the tracking error can be reduced by 61.11%.