Rational design and fabrication of efficient catalysts for overall water splitting in an integrated electrolyzer are essential for energy storage and conversion. Herein, an ultra‐small substitutional Zn‐doped Ru/RuO2 heterostructure (5.6 nm) is synthesized through a facile pyrolyzing strategy as an efficient bi‐functional electrocatalyst for water splitting in both acidic and alkaline media. Experiments demonstrate that introducing appropriate alien atoms into RuO2 with an amorphous state can effectively optimize the electron structure and expose abundant defects, and thus increasing the number of active sites and improving the intrinsic activity of RuO2 for the oxygen evolution reaction (OER). In addition, the high hydrogen adsorption capability of metallic Ru makes Ru‐RuO2 heterointerface suitable for water splitting. Notably, the Ru/ZnRuO2 displays low overpotentials at 10 mA cm−2 with only 184 mV for OER in 0.5 m H2SO4 and 35 mV for HER in 1.0 m KOH. Besides, the as‐synthesized Ru/ZnRuO2 displays cell voltages of 1.540 and 1.567 V (at 10 mA cm−2) for water splitting with remarkable durability for more than 220 and 100 h, respectively, in acidic and alkaline media. This work provides a facile strategy for designing pH‐universal electrocatalysts toward overall water splitting.