Herein, the formation of the vortex and slag entrainment in tundish is studied by using mathematical and physical simulation. This article aims at contributing to the understanding of the free surface vortex formation in continuous casting tundish at the end of casting. A 3D multiphase mathematical model using the renormalization group (RNG) k–ε turbulence model is developed to predict the vortex formation and slag layer behavior on the scale of the 1:4 water model of a two‐strand slab tundish. The Eulerian volume of fluid model is adopted to track the steel–slag–air free surface and vortex, and the effect of Coriolis force is considered in this simulation. An improved vortex prevention design is also proposed by changing the position of the diversion hole on the dam of the tundish, which is compared with the original scheme using numerical and physical simulation. The predicted results of the vortex structure and the critical height of the vortex formation agree well with the water modeling results. It is demonstrated that the critical height of the vortex formation is reduced significantly by regulating the position of the diversion hole on the dam near the outlet of the tundish.