Due to the variety of background model in the real world, detecting changes in a video cannot be addressed exhaustively by a simple background subtraction method, especially with several motion detection challenges, such as dynamic background, camera jitter, intermittent object motion, and so on. In this paper, we propose an efficient background subtraction method, namely locally statistical dual-mode (LSD), for detecting moving objects in video-based surveillance systems. The method includes a local intensity pattern comparison algorithm for foreground segmentation by analyzing the homogeneity of intensity patterns of the input frame and the background model, in which the homogeneity is calculated by the mean and standard deviation of pixel intensity. Besides that, a dual-mode scheme is developed to temporally update the background model for the short-and long-term scenarios corresponding to sudden and gradual changes in the background. The advantage of this scheme is the allowance of updating the model in both pixel-and framewise manners simultaneously. The parameters used in both the local intensity pattern comparison algorithm and the dual-mode background model updating scheme are estimated for every input frame consecutively based on local and global statistical information of segmentation result. In experiments, the proposed LSD method is extensively evaluated on the Wallflower and CDnet2014 datasets; and remarkable performance demonstrates its preeminence to the many state-of-the-art background subtraction approaches in terms of segmentation accuracy and computational complexity. INDEX TERMS Motion detection, background subtraction, background modeling, moving object detection, video segmentation, locally statistical dual-mode updating. THIEN HUYNH-THE (S'15-M'19) received the B.S. degree in electronics and telecommunication engineering and the M.Sc. degree in electronics engineering from the