Since ferroelectricity has been observed in simple binary oxide material systems, it has attracted great interest in semiconductor research fields such as advanced logic transistors, non-volatile memories, and neuromorphic devices. The location in which the ferroelectric devices are implemented depends on the specific application, so the process constraints required for device fabrication may be different. In this study, we investigate the ferroelectric characteristics of Zr doped HfO2 layers treated at high temperatures. A single HfZrOx layer deposited by sputtering exhibits polarization switching after annealing at a temperature of 850 °C. However, the achieved ferroelectric properties are vulnerable to voltage stress and higher annealing temperature, resulting in switching instability. Therefore, we introduce an ultrathin 1-nm-thick Al2O3 layer at both interfaces of the HfZrOx. The trilayer Al2O3/HfZrOx/Al2O3 structure allows switching parameters such as remnant and saturation polarizations to be immune to sweeping voltage and pulse cycling. Our results reveal that the trilayer not only makes the ferroelectric phase involved in the switching free from pinning, but also preserves the phase even at high annealing temperature. Simultaneously, the ferroelectric switching can be improved by preventing leakage charge.