The polarity of the resistive switching (RS) characteristic of metal‐oxide‐metal devices from atomic layer deposited polycrystalline ZrO2 films was studied by means of impedance spectroscopy. Pt/ZrO2/Ti/Pt cells made with 10 nm Ti and 30 nm Pt capping top electrodes, served as unipolar switching (US) devices. Bipolar switching (BS) devices were represented by Pt/ZrO2/30 nm TiN cells. Temperature measurements of the ON‐state resistances clearly show metallic and semiconducting behavior for the US and BS cells, respectively. The pristine and the ON and OFF states of the devices were analyzed by means of impedance spectroscopy. All ZrO2 based RS devices exhibited similar impedance characteristics in the pristine states. In contrast, after electroforming clear differences in the Nyquist‐plots of the US and BS devices were observed. The effect of the device structure on the RS polarity is discussed under consideration of the pillar‐shaped grainy microstructure of the ZrO2 thin films. An empirical model based on redox reactions between ZrO2 and the non‐noble metal electrode is proposed emphasizing defect formation prior at the ZrO2 grain boundaries.