Annealing at moderate temperatures is required to activate the silicon surface passivation by Al2O3 thin films while also the thermal stability at higher temperatures is important when Al2O3 is implemented in solar cells with screenprinted metallization. In this paper, the relationship between the microstructure of the Al2O3 film, hydrogen diffusion, and defect passivation is explored in detail for a wide range of annealing temperatures. The chemical passivation was studied using stacks of thermally-grown SiO2 and Al2O3 synthesized by atomic layer deposition. Thermal effusion measurements of hydrogen and implanted He and Ne atoms were used to elucidate the role of hydrogen during annealing. We show that the passivation properties were strongly dependent on the annealing temperature and time and were significantly influenced by the Al2O3 microstructure. The latter was tailored by variation of the deposition temperature (Tdep = 50 °C–400 °C) with hydrogen concentration [H] between 1 and 13 at.% and mass density ρmass between 2.7 and 3.2 g/cm3. In contrast to films with intermediate material properties, the passivation by low- and high density films showed a reduced thermal stability at relatively high annealing temperatures (∼600 °C). These observations proved to be in good agreement with thermal effusion results of hydrogen and inert gas atoms that were also strongly dependent on film microstructure. We demonstrate that the temperature of maximum effusion decreased for films with progressively lower density (i.e., with increasing [H]). Therefore, the reduced thermal stability of the passivation for low-density hydrogen-rich ([H] >∼5 at. %) films can be attributed to a loss of hydrogen at relatively low annealing temperatures. In contrast, the lower initial [H] for dense Al2O3 films can likely explain the lower thermal stability associated with these films. The effusion measurements also allowed us to discuss the role of molecular- and atomic hydrogen during annealing.