Non-uniform blind deblurring of dynamic scenes has always been a challenging problem in image processing because of the diverse of blurring sources. Traditional methods based on energy minimization cannot make accurate kernel estimation. It leads to that some high frequency details cannot be fully recovered. Recently, many methods based on convolution neural networks (CNNs) have been proposed to improve the overall performance. Followed by this trend, we first propose a two-stage deblurring module to recover the blur images of dynamic scenes based on high frequency residual image learning. The first stage performs initial deburring with the blur kernel estimated by the salient structure. The second stage calculates the difference of input image and initially deblurred image, referred to as residual image, and adopt an encoder-decoder network to refine the residual image. Finally, we can combine the refined residual image with the input blurred image to obtain the latent image. To increase deblurring performance, we further propose a coarse-to-fine framework based on the deblurring module. It performs the deblurring module many times in a multi-scale manner which can gradually restore the sharp edge details of different scales. Experiments conducted on three benchmark datasets demonstrate the proposed method achieves competitive performance of state-of-art methods.INDEX TERMS Image deblurring, dynamic blur, non-uniform blind deblurring, deep learning.