It is widely accepted that the rubber concrete (RC) originating from waste is a promising material that can contribute to the conservation and rational use of natural resources and the protection of the environment. However, the fatigue performance in a hygrothermal environment is a major concern because little pertinent information is available in the relevant literature. In this study, a cyclic loading test was carried out on RC subjected to different wet-dry cycles at different temperatures. The loading strain, plastic strain, and elastic strain of the concrete were compared and analyzed. The results revealed that the loading strain and plastic strain of the RC were obvious after the 1st loading cycle. As the number of loading cycles increased, the stress-strain curve became denser and the RC exhibited good elasticity. As the wet-dry cycles increased, the average plastic strain in the 10th–60th loading cycle increased while the elastic strain decreased. After 28 wet-dry cycles, the average plastic strain at 60°C increased by 42.31% compared with 20°C. In fact, as the temperature became higher, the plastic damage incurred by the RC became more severe. Finally, the damage variable was defined based on the elastic modulus and plastic strain to evaluate the fatigue performance of the RC in a hygrothermal environment. The findings of this study can provide a useful reference for RC applications.