Dust is a common air pollution source. The color of images captured under dusty weather is usually yellow even brown, which reduces scene visibility and causes the loss of image details. To remove the dust and make image scene clear, this article presents a simple and effective image dedusting network called FFNet. In the process of image feature extraction, the FFNet uses several residual blocks with smoothed dilated convolution, that is, common dilated convolution followed by separable and shared (SS) blockwise fully connected operation to extend the receptive field and reduce gridding artifacts caused by common dilated convolution. Furthermore, the FFNet fuses image features from different layers via an adaptive weighting scheme. Due to the difficulty of collecting real dusty images, we used our proposed dusty image synthesis scheme to achieve data augmentation for better network training. Experiments on a series of synthetic and real dusty images demonstrate that the FFNet obtains better image dedusting performance than several state-of-the-art image restoration methods.