The mesh stiffness characteristics of the gear pair have direct effect on its transmission error, and then affect the noise and vibration level of the gear. To control the instability of gear meshing caused by tooth deformation, the tooth tip modification is an important mean, but it will undoubtedly introduce tooth profile errors. Firstly, considering the inconsistency between base circle and root circle under different tooth numbers, the potential energy method is improved to calculate the mesh stiffness of spur gear pair. Then, based on the theory of tooth tip modification and improved potential energy method, a new analytical method for mesh stiffness calculation considering tooth profile error is proposed. Taking a pair of spur gear as example, the finite element(FE) model is established, and the correctness of the proposed method is verified by comparing the time-varying mesh stiffness, mean stiffness, and peak-to-peak stiffness. The comparison results show that the method proposed in this paper can ensure high calculation accuracy while achieve lower computation cost. Finally, the sensitivity analysis is conducted on modification parameters including curve types, amount, and length. The influence of different parameters on mesh stiffness characteristics, bearing capacity, and noise problem is investigated.