Ferroelectric materials offer a low-energy, high-speed alternative to conventional logic and memory circuitry. Hafnia-based films have achieved singledigit nm ferroelectricity, enabling further device miniaturization. However, they can exhibit nonideal behavior, specifically wake-up and fatigue effects, leading to unpredictable performance variation over consecutive electronic switching cycles, preventing large-scale commercialization. The origins are still under debate. Using plasmon-enhanced spectroscopy, a non-destructive technique sensitive to <1% oxygen vacancy variation, phase changes, and single switching cycle resolution, the first real-time in operando nanoscale direct tracking of oxygen vacancy migration in 5 nm hafnium zirconium oxide during a pre-wake-up stage is provided. It is shown that the pre-wake-up leads to a structural phase change from monoclinic to orthorhombic phase, which further determines the device wake-up. Further migration of oxygen ions in the phase changed material is then observed, producing device fatigue. These results provide a comprehensive explanation for the wake-up and fatigue with Raman, photoluminescence and darkfield spectroscopy, combined with density functional theory and finite-difference time-domain simulations.