In this paper, we develop a finite difference method for solving the wave equation with fractional damping in 1D and 2D cases, where the fractional damping is given based on the Caputo fractional derivative. Firstly, based on the weighted method, we propose a new numerical approximation for the Caputo fractional derivative and apply it for the 1D case to obtain a time-stepping method. We then develop an alternating direction implicit (ADI) scheme for the 2D case. Using the discrete energy method, we prove that the proposed difference schemes are unconditionally stable and convergent in both 1D and 2D cases. Finally, several numerical examples are given to verify the theoretical results.