This study investigates the operating characteristics of AlGaN/GaN high-electron-mobility transistors (HEMTs) by applying HfO2 as the passivation layer. Before analyzing HEMTs with various passivation structures, modeling parameters were derived from the measured data of fabricated HEMT with Si3N4 passivation to ensure the reliability of the simulation. Subsequently, we proposed new structures by dividing the single Si3N4 passivation into a bilayer (first and second) and applying HfO2 to the bilayer and first passivation layer only. Ultimately, we analyzed and compared the operational characteristics of the HEMTs considering the basic Si3N4, only HfO2, and HfO2/Si3N4 (hybrid) as passivation layers. The breakdown voltage of the AlGaN/GaN HEMT having only HfO2 passivation was improved by up to 19%, compared to the basic Si3N4 passivation structure, but the frequency characteristics deteriorated. In order to compensate for the degraded RF characteristics, we modified the second Si3N4 passivation thickness of the hybrid passivation structure from 150 nm to 450 nm. We confirmed that the hybrid passivation structure with 350-nm-thick second Si3N4 passivation not only improves the breakdown voltage by 15% but also secures RF performance. Consequently, Johnson’s figure-of-merit, which is commonly used to judge RF performance, was improved by up to 5% compared to the basic Si3N4 passivation structure.