Image dehazing on a hazy image aims to remove the haze and make the image scene clear, which attracts more and more research interests in recent years. Most existing image dehazing methods use a classic atmospheric scattering model and natural image priors to remove the image haze. In this paper, we propose an end-to-end image dehazing model termed as DRHNet (Deep Residual Haze Network), which restores the haze-free image by subtracting the learned negative residual map from the hazy image. Specifically, DRHNet proposes a context-aware feature extraction module to aggregate the contextual information effectively. Furthermore, it proposes a novel nonlinear activation function termed as RPReLU (Reverse Parametric Rectified Linear Unit) to improve its representation ability and to accelerate its convergence. Extensive experiments demonstrate that DRHNet outperforms state-of-the-art methods both quantitatively and qualitatively. In addition, experiments on image deraining task show that DRHNet can also serve for image deraining. INDEX TERMS Image dehazing, image deraining, negative residual map, context-aware feature extraction, reverse parametric rectified linear unit (RPReLU).