This study focuses on the tribological characteristics of polytetrafluoroethylene rubbing against 316L stainless steel in the presence of various Al2O3 abrasive particles. The ranges of the contact stresses and abrasive sizes were 2.50–5.01 MPa and 2.0–230 µm, respectively. The evolution of the coefficient of friction, abrasive invasion behavior and damage characteristics on worn surfaces, and the transformation of damage mechanisms under different contact stresses are summarized. Results demonstrate that contact stress exerts a considerable effect on abrasive invasion behavior, the coefficient of friction trends become complicated at low contact stress. With decreasing particle sizes under a contact stress of 2.50 MPa, the wear failure changes from two-body wear to three-body wear, or two wear behaviors exist simultaneously. As contact stress increases, the wear mechanism is gradually simplified. The particle size threshold that causes the change of the wear mechanism is also gradually reduced. The wear rate of polytetrafluoroethylene has no linear relationship with abrasive size, whereas wear increases with contact stress. Under 2.50 MPa conditions, the particles easily invade the tribo-interface in the friction process, thus increasing the wear rate of the metallic counterpart. In addition, with increases contact stress, the polytetrafluoroethylene material transfers to the counterface, and local adhesion occurs. When the contact stress is increased to 5.01 MPa, polytetrafluoroethylene even shows an obvious creep.