The steady and unsteady leakage flow and heat transfer characteristics of the rotor blade squealer tip were conducted by solving Reynolds-Averaged Navier-Stokes (RANS) equations with k-ω turbulence model. The first stage of GE-E3 engine with squealer tip in the rotor was adopted to perform this work. The tip clearance was set to be 1% of the rotor blade height and the groove depth was specified as 2% of the span. The results showed that there were two vortexes in the tip gap which determined the local heat transfer characteristics. In the steady flow field, the high heat transfer coefficient existed at several positions. In the unsteady case, the flow field in the squealer tip was mainly influenced by the upstream wake and the interaction of the blades potential fields. These unsteady effects induced the periodic variation of the leakage flow and the vortexes, which resulted in the fluctuation of the heat transfer coefficient. The largest fluctuation of the heat transfer coefficient on the surface of the groove bottom exceeded 16% of the averaged value on the surface of the squealer tip.