Early detection of people’s suspicious behaviors can aid in the prevention of crimes and make the community safer. Existing methods that are focused on identifying abnormal behaviors from video surveillance that are based on computer vision, which are more suitable for detecting ongoing behaviors. While criminals intend to avoid abnormal behaviors under surveillance, their suspicious behaviors prior to crimes will be unconsciously reflected in the trajectories. Herein, we characterize several suspicious behaviors from unusual movement patterns, unusual behaviors, and unusual gatherings of people, and analyze their features that are hidden in the trajectory data. Meanwhile, the algorithms for suspicious behavior detection are proposed based on the main features of the corresponding behavior, which employ spatiotemporal clustering, semantic annotation, outlier detection, and other methods. A practical trajectory dataset (i.e., TucityLife) containing more than 1000 suspicious behaviors was collected, and experiments were conducted to verify the effectiveness of the proposed method. The results indicate that the proposed method for suspicious behavior detection has a recall of 93.5% and a precision of 87.6%, demonstrating its excellent performance in identifying the possible offenders and potential target places. The proposed methods are valuable for preventing city crime and supporting the appropriate allocation of police resources.