Rubbers are widely used in various fields as the important sealing materials, such as window seal, door seal, valve, pump seal, etc. The fretting wear behavior of rubbers has an important effect on their sealing performance. This paper presents an experimental study on the fretting wear behavior of rubbers against the steel ball under air conditions (room temperature at 20 ± 2 °C and humidity at 40%). Three kinds of rubbers, including EPDM (ethylene propylene diene monomer), FPM (fluororubber), and NBR (nitrile–butadiene rubber), are considered in experiments. The sphere-on-flat contact pattern is used as the contact model. The influences of the displacement amplitude, normal force, frequency, and rubber hardness on the fretting wear behavior are discussed in detail. White light profiler and scanning electron microscope (SEM) are used to analyze the wear mechanism of the rubber surface. The fretting wear performances of three rubbers are compared by considering the effect of the displacement amplitude, normal force, frequency, and rubber hardness. The results show that NBR has the most stable friction coefficient and the best wear resistance among the three rubbers.