As a core step of obstacle avoidance and path planning, dynamic obstacle detection is critical for autonomous driving. This study aimed to propose a dynamic obstacle detection method based on U–V disparity and residual optical flow for autonomous driving. First, a drivable area of an unmanned vehicle was detected using U–V disparity images. Then, obstacles in the drivable area were detected using U–V disparity images and the geometric relationship between obstacle size and its disparity. Finally, the motion likelihood of each obstacle was estimated by compensating the camera ego-motion. The innovation of the proposed method was that the searching range of the moving obstacles was greatly narrowed by detecting the obstacles in the drivable area, which greatly improved not only the moving obstacle detection efficiency but also the detection accuracy. Datasets from the KITTI benchmark and our self-acquired campus scene data were chosen as testing samples. The experimental results showed that our method could achieve high detection precision, low missed detection rate and less time consumption.