To reduce water traffic accident, identification of overloaded ships has been of major importance for enforcing inland river management, especially in some developing countries. Driven by rising oil prices or other economic benefits, the ship overload phenomenon continued to occur in China. Therefore, overloaded ship detection has been of a key factor of shipping safety. This paper presented a robust method for detecting overloaded ship and the proposed algorithm included three stages: ship detection, ship tracking and overloaded ship identification. Ship detection was a key step and the concept of ship tracking is built upon the ship-segmentation method, in which the algorithm about background estimation, background updating, background subtraction and ship detection has been described. According to the segmented ship shape, a predict method based on Kalman filter has been proposed to track each ship. The described identification system included a video camera and a high definition camera, which led to a necessary coordinate transformation in system model of Kalman filter. The data of ship length and ship speed could be used to identify overloaded ship. The proposed method has been tested on a number of monocular ship image sequences and the experimental results showed that the algorithm was robust and real-time.