Atomic layer deposition was used to deposit amorphous Er-doped Al2O3 films (0.9–6.2 at. % Er) on Si(100). The Er3+ photoluminescence (PL), Er3+ upconversion luminescence, as well as the Si PL and associated surface passivation properties of the films were studied and related to the structural change of the material during annealing. The PL signals from Er3+ and Si were strongly dependent on the annealing temperature (T = 450–1000 °C), but not directly influenced by the transition from an amorphous to a crystalline phase at T > 900 °C. For T > 650 °C, broad Er3+ PL centered at 1.54 μm (4I13/2) with a full width at half maximum of 55 nm was observed under excitation of 532 nm light. The PL signal reached a maximum for Er concentrations in the range of 2–3 at. %. Multiple photon upconversion luminescence was detected at 660 nm (4F9/2), 810 nm (4I9/2), and 980 nm (4I11/2), under excitation of 1480 nm light. The optical activation of Er3+ was related to the removal of quenching impurities, such as OH (3 at. % H present initially) as also indicated by thermal effusion experiments. In contrast to the Er3+ PL signal, the Si luminescence, and consequently the Si surface passivation, decreased for increasing annealing temperatures. This trade-off between surface passivation quality and Er3+ PL can be attributed to an opposite correlation with the decreasing hydrogen content in the films during thermal treatment.