Experimental method for a characterization of high-cycle fatigue evolution of residual stress near cold-expanded hole is developed and implemented. The technique is based on simultaneous measurements of deformation response to narrow notch, inserted in residual stress field, on opposite specimen’s faces by electronic speckle-pattern interferometry (ESPI). Two-side measurements of notch opening displacements are performed when a single notch, emanating from cold-expanded hole edge, is inserted. The transition from in-plane displacement component to residual stress intensity factor (SIF) values follows from the relationships of modified version of the crack compliance method. The approach provides a difference in residual stress values referred to mandrel entrance and exit surface. Notches are inserted at different stages of low-cycle fatigue without applying external load. The results obtained describe fine nuances of residual stress evolution, which cannot be considered as monotonic relaxation.