Meshing power loss is one of the most important parts in power loss calculation of planetary gear set. However, most of the conventional methods assumed the friction coefficient between gears as a constant value in the meshing power loss calculation, and most importantly, the influence of gear tooth surface geometry is usually ignored, for example, roughness. Therefore, a new meshing power loss calculation model for planetary gear set that considers tooth surface roughness is proposed on the basis of elasto-hydrodynamic lubrication method. With the proposed model, a planetary gear set dynamic model that considers friction force between gears is first established to study the time-varying meshing forces, sliding speeds, and curvature radii of the gear pairs. Then, an elasto-hydrodynamic lubrication model of the gear pair contact interface is constructed to investigate and modify the friction force distribution in the gear meshing process of the dynamic model iteratively until the meshing forces converge to stable values. Furthermore, the relationship between the tooth surface roughness and film thickness is studied in the elasto-hydrodynamic lubrication model. After that, the meshing power loss is calculated based on the obtained meshing forces, friction coefficients, sliding speeds, and so on. The results show that there is a sudden growth of the meshing power loss at the end of the meshing cycle, which has a good agreement with the meshing force impact. And, it is found that tooth surface roughness has a direct influence on the meshing power loss of sun–planet gear pair, which yields an increasing tendency as the surface roughness growing.