The radiation effects of electrons and protons on the spectroscopic and optical properties of oxide glasses doped with Yb3+ in various glass systems were investigated to understand the impact of the glass composition on the glass photo-response. Changes in the optical and emission properties were seen after the radiation treatment, and the magnitude of the changes depended on the irradiation source and dose. For all the investigated materials, the absorption coefficients in the 200–550 nm range increase post-irradiation, revealing the formation of defects in the glasses during the irradiation. While the spectroscopic properties of the tellurite glass remain unchanged, a small reduction in the Yb3+ emission intensity was seen after irradiating the phosphate, borosilicate, and germanate glasses, indicating that a reduction of Yb3+ to Yb2+ might occur in these glasses during the radiation treatment. The changes in the optical and spectroscopic properties after proton irradiation are small as they are localized at the surface of the glasses due to the shallow penetration depth of the proton in the glass. Even though the doses are small, the electron irradiation produces larger changes in the optical and spectroscopic properties since the electrons penetrate the entire volume of the glasses. All the changes in the optical and spectroscopic properties of the glasses were successfully reversed after a short heat treatment revealing the reversible nature of the photo-response of the investigated glasses.