In this paper, we introduce a detailed study of degradation of the In x Ga 1-x As quantum well (QW) based semiconductor saturable absorber mirror (SESAM) irradiated with picosecond pulses from a fiber laser at a 1064 nm wavelength. In a slow degradation study at a lower incident fluence, the longest operating times of ~4000 h were recorded for two SESAMs with a slow carrier relaxation (τ = 15 ps) in comparison to 1000 h of the fast (τ = 1 ps) SESAM. The nonlinear reflectivity measurements after the long-term tests indicated that the most likely mechanism of SESAM degradation was the modification of the InGaAs QW structure. By reducing a thermal load on the SESAM, an expected lifetime was increased tenfold. At higher saturation levels, SESAMs experienced critical optical damage (COD) after less than 24 h of operation. The most likely mechanism of COD was deduced to be the two-photon absorption in GaAs material. The necessity of slow degradation tests close to the operating conditions of the SESAM was demonstrated.