The unknown mutual coupling effect between antennas significantly degrades the target localization performance in the bistatic multiple-input multiple-output (MIMO) radar. In this paper, the joint estimation problem for the direction of departure (DOD) and direction of arrival (DOA) is addressed. By exploiting the target sparsity in the spatial domain and formulating a dictionary matrix with discretizing the DOD/DOA into grids, compressed sensing (CS)-based system model is given. However, in the practical MIMO radar systems, the target cannot be precisely on the grids, and the unknown mutual coupling effect degrades the estimation performance. Therefore, a novel CS-based DOD/DOA estimation model with both the off-grid and mutual coupling effect is proposed, and a novel sparse reconstruction method is proposed to estimate DOD/DOA with updating both the off-grid and mutual coupling parameters iteratively. Moreover, to describe the estimation performance, the corresponding Cramér–Rao lower bounds (CRLBs) with all the unknown parameters are theoretically derived. Simulation results show that the proposed method can improve the DOD/DOA estimation in the scenario with unknown mutual coupling effect, and outperform state-of-the-art methods.