Rubber O-rings are widely applied in the static and dynamic seals of machinery, energy, chemical, aviation, and other fields. Mechanical behavior and sealing performance of the O-ring were investigated in this paper. Effects of precompression amount, fluid pressure, friction coefficient on the static and dynamic sealing performances of the O-ring were studied. The results show that the maximum stress appears on the inside but not surface of the O-ring. The static sealing performance increases with the increasing of fluid pressure and compression amount. Reciprocating dynamic sealing performance of the rubber O-ring is different with its static sealing performance; the stress distribution and deformation are changing in reciprocating motion. Sealing performance in outward stroke is better than it in the inward stroke. Overturn of the O-ring occurs when the friction torque is greater than the torque that caused by fluid pressure in the inward stroke. Distortion, bitten, and fatigue failure are the main failure modes of the O-ring in the dynamic seal. Those results can be used in design, installation, and operation of rubber O-rings in static and dynamic seals.