In the reciprocating labyrinth piston compressor, the characteristic of the internal leakage is crucial for the leakage management and performance improvement of the compressor. However, most of the published studies investigated the rotor-stator system, and those who study the reciprocating piston-cylinder system basically focus on the effects of the geometrical parameters. These conclusions could not directly be applied to predict the real-time leakage flow rate through the labyrinth seal because of the fast reciprocating motion of the piston, which will cause continually pressure change in two compression chambers, and then the pressure fluctuation will affect the flow through the labyrinth seal. A transient simulation model employing the multiscale dynamic mesh, which considers the effect of the reciprocating motion of the piston in the cylinder, is established to identify the characteristics of the internal leakage. This model was verified by a specially designed compressor, and the influence of various parameters was analyzed in detail. The sealing performance decreased linearly with the increase in the pressure ratio, and higher pressure inlet leads to higher leakage flow under the same pressure ratio. The labyrinth seal performance positively correlated to the increase of the rotational speed. Leakage characteristics of five working mediums were carried out, and the results indicated that the relative leakage decreased with an increase in the relative molecular mass. From this study, the realistic internal leakage flow rate under different operating parameters in the reciprocating labyrinth piston compressor could be predicated.