The effective case depth (ECD) plays an important role in the meshing strength of internal gear transmissions. Carburizing quenching heat treatment is commonly used to enhance gear strength and wear resistance. However, the different ECDs in internal and external gears caused by heat treatment significantly affect the meshing strength, causing vibration, reducing gear service life, and hastening malfunction in internal gear transmission. In this study, we conducted an investigation of different ECDs by the heat treatment of carburized gear pairs by numerical simulation with the finite element method (FEM) and experiment tests. We analyzed three different carburized layer models, with the ECD in the internal gear being greater than, less than, and equal to the ECD in the external gear. In addition, we investigated the ability to distinguish between hardness gradients in gear teeth by dividing the carburized depth into seven layers to improve modeling accuracy. Results revealed that the meshing strength of internal gear transmission could be significantly enhanced by adopting the model with the ECD in the internal gear being less than the ECD in the external gear, and moreover, the shear stress of carburized gears initially increased and then decreased along with depth direction, and the maximum value appeared in the middle of the lower surface.