The strong oxidant H2O2 is known to exist in solid form on Europa and is suspected to exist on several other Solar System worlds at temperatures below 200 K. However, little is known of the thermal chemistry that H2O2 might induce under these conditions. Here, we report new laboratory results on the reactivity of solid H2O2 with eight different compounds in H2O-rich ices. Using infrared spectroscopy, we monitored compositional changes in ice mixtures during warming. The compounds CH4 (methane), C3H4 (propyne), CH3OH (methanol), and CH3CN (acetonitrile) were unaltered by the presence of H2O2 in ices, showing that exposure to either solid H2O2 or frozen H2O+H2O2 at cryogenic temperatures will not oxidize these organics, much less convert them to CO2. This contrasts strongly with the much greater reactivity of organics with H2O2 at higher temperatures, and particularly in the liquid and gas phases. Of the four inorganic compounds studied, CO, H2S, NH3, and SO2, only the last two reacted in ices containing H2O2, NH3 making NH4+ and SO2 making SO(4)2- by H+ and e- transfer, respectively. An important astrobiological conclusion is that formation of surface H2O2 on Europa and that molecule's downward movement with H2O-ice do not necessarily mean that all organics encountered in icy subsurface regions will be destroyed by H2O2 oxidation.