In marine transportation, most ships are equipped with AIS devices. The AIS data sent by these devices can help maritime authorities to manage ships in relevant sea areas. However, AIS is a self-reporting system, when a ship is engaged in illegal activities, the AIS device may be turned off. Therefore, after the AIS is closed, if the ship's behavior during a certain period of time is different from the ship's behavior before the closure of AIS, the different behavior is likely to represent that the ship is conducting illegal activities. This behavior is considered abnormal and needs to be detected in time. Based on radar trajectory data, the detection of abnormal ship behavior is studied from two aspects: speed and direction. In order to improve the intelligent level of abnormal ship behavior detection, the abnormal speed behavior detection algorithm combined with rules and clustering (ASBD-RC) and the abnormal direction behavior detection algorithm combined with partition and the earth mover's distance (ADBD-PE) are proposed. The ASBD-RC algorithm can reduce the influence of noise and sea clutter on abnormal speed behavior detection. The ADBD-PE algorithm can effectively partition and identify trajectory segments with abnormal direction. In the experiment, based on the real and simulated radar trajectories, the abnormal behaviors of ships under different scenarios are generated. The experimental results show that in most scenarios, the ASBD-RC algorithm and the ADBD-PE algorithm can effectively detect abnormal ship behavior. And compared with other algorithms, the proposed two algorithms have better and more stable detection results.