Rotor seals are the most important element of turbomachine. Seals characteristics determine efficiency, reliability and lifetime of turbopump. Labyrinth seals have lower flow coefficient compared to gap seals. However they are used not so often because under certain conditions they may cause of rotor dynamic instability. Vortex in seal grooves are one of the causes of rotor oscillations. And their number, configuration, dimensions and velocity directly depend on groove geometry. The paper describes the modeling of the flow in labyrinth seal using ANSYS CFX, the influence of groove geometry on vortex number, dimension and configuration.