An experimental and numerical investigation into the tip leakage flow of a turbine rotor is carried out using a particle image velocimetry (PIV) system and the commercial software ANSYS CFX 14.0. The specimen used in this work is a typical GE-E 3 model with a new squealer tip design. The experimental data are used to create a turbulence model and numerical strategy. Through the validated turbulence model and numerical strategy, simulations are carried out to compare the characteristics of the tip leakage flow in three cases: (1) the blade is rotating, but the shroud is stationary, which is the real status of turbine rotor operation; (2) the blade is stationary, but the shroud moves, to simulate their relative movement; (3) the blade is stationary, and the shroud is also stationary, this is a simplified case, but has been widely used in the experiments on rotor tip leakage flow. Detailed analysis of the flow phenomena shows that the second case is a reasonable alternative approach to simulate the real state. However, the flow patterns in the third case exhibit some evident differences from the real status. These differences are caused by the inaccurate viscous force arising from the stationary blade and shroud. In this work, a modification method for the experiments conducted in the third case is firstly proposed, which is realized through adding an imaginary roughness at the shroud wall to be close to the real viscous effect, and to thereby reduce the deviation of the experiment from the real case. According to the results calculated by ANSYS CFX, the flow structure in the modification case is very close to the real status. Besides, this modification case is an easy and cheap way to simulate the real tip leakage flow.