The novel destructive method is implemented for quantitative assessment of fatigue damage accumulation in the stress concentration zone accompanied by residual stress due to cold expansion of the through-thickness hole. Damage accumulation is reached by preliminary cyclic loading of plane specimens with cold-expanded holes. Narrow notches, emanating from the hole edge at different stages of high-cycle fatigue, serve to manifest a damage level. These notches are inserted without applying external load. Deformation response to local material removing, caused by pure residual stress influence, is measured by electronic speckle pattern interferometry (ESPI) in terms of in-plane displacement components. Normalized values of the notch mouth open displacement (NMOD), in-plane displacement component at the initial point of the notch acting in the notch direction (U0), in-plane displacement component at the final point of the notch acting in the notch direction (U1) and the stress intensity factor (SIF) are used as current damage indicators. Numerical integration of curves, describing an evolution of each fracture mechanics parameter over lifetime, produces the damage accumulation function in an explicit form. It is established that all four fracture mechanics parameters give very close results.