In order to optimize sealing performance, a novel labyrinth seal with semi-elliptical teeth (SET) structure is proposed in this paper, which includes semi-elliptical teeth and a series of cavities. The simulation results calculated by the numerical methods are compared with the experimental and theoretical results, and static and dynamic characteristics of the novel SET structure are further investigated. The numerical simulations of labyrinth seals with the SET structure demonstrate high accuracy and reliability, with a maximum relative error of less than 6% as compared to experimental results, underscoring the validity of the model. Notably, leakage rates are directly influenced by pressure drop and axial offset, with optimal sealing achieved at zero axial displacement. The direct damping coefficient increases as the pressure drop increases while the other dynamic coefficients decrease. Additionally, the stability results show that the novel SET structure exhibits higher stability for positive axial offsets. The novel model and corresponding results can provide a meaningful reference for the study of sealing structure and coupled vibration in the field of fluid machinery.