Reversible data hiding (RDH) algorithm is developing rapidly for many years. Usually, the histogram shifting (HS) based RDH algorithm has two main steps: Firstly, a steep difference histogram is generated with some effective methods for difference. Secondly, by extending and shifting some differences in the histogram, bits of information can be embedded into the cover image which can be restored reversibly. In this paper, we proposed the second-order difference to obtain a steeper difference histogram. Firstly, we slide a window with size 2 × 2 through the image. For each block, we can get the two first-order differences by calculating the absolute values of the two differences for its two columns. Thus, the secondorder difference of each block, which is the absolute values of the difference of the two first-order differences, can be obtained. By extending and shifting the second-order difference, a bit may be embedded into the block finally. Experiments reveal that the proposed algorithm outperforms the previous state-of-the-art RDH methods in terms of the computational complexity, image distortion and the embedding performance. INDEX TERMS Difference expanding, cover image, histogram shifting, reversible data hiding, the secondorder difference, watermarking.