Abstract. We report on the dynamic behavior of a rotary lip seal by considering the interaction between lip, film and shaft roughness assumed to have a periodic form. The nonlinearities of stiffness and viscosity of the film are taken into account in a mass-spring-dumper model. Using the harmonic balance method, analytical prediction of the lip displacement is obtained, the frequency response is provided and the effect of the shaft undulation on the amplitude jumps of the lip displacement and on the film thickness fluctuations are discussed. The results have direct applications in reducing leakage that may occur between a smooth lip seal and a rough shaft.