The second-order nonlinear-polarization originated from the interaction between thin-film material with second-order nonlinear susceptibility (χ(2)
) and high-power laser is essential for integrated optics and photonics. In this work, strong second-order nonlinear-polarization was found in a-axis oriented Zn1-xMgxO (ZnMgO) epitaxial thin-films with Li incorporation, which were deposited by radio-frequency magnetron sputtering. Mg incorporation (x>0.3) causes a sharp fall in the matrix element χ33
of χ(2)
tensor, although it widens optical bandgap (Eopt
). In contrast, moderate Li incorporation significantly improves χ33
and resistance to high-power laser pulses with a little influence on Eopt
. In particular, a Zn0.67Mg0.33O:Li [Li/(Zn+Mg+Li)=0.07] thin-film shows a |χ33
| of 36.1 pm/V under a peak power density (Ep
) of 81.2 GW/cm2, a resistance to laser pulses with Ep
of up to 124.9 GW/cm2, and an Eopt
of 3.95 eV. Compared to that of ZnO, these parameters increase by 37.8%, 53.4%, and 18.6%, respectively. Specially, the Zn0.67Mg0.33O:Li shows higher radiation resistance than a Mg-doped LiNbO3 crystal with a comparable Eopt
. First-principle calculations reveal the Li occupation at octahedral interstitial sites of wurtzite ZnO enhances radiation resistance by improving structural stability. X-ray photoelectron spectroscopy characterizations suggest moderate Li incorporation increases χ33
via enhancing electronic polarization. These findings uncover the close relationship between the octahedra interstitial defects in wurtzite ZnMgO and its nonlinear-polarization behavior under the optical frequency electric field of high-power laser.