Three composition groups in the BaTiO3–CaTiO3–BaHfO3 ternary system, (1 − x)Ba(HfyTi1−y)O3–x(Ba1−zCaz)TiO3 (abbreviated as BHyT–xBCzT with x = 0–1.0, y = 0.16, z = 0.20; x = 0–1.0, y = 0.16, z = 0.30; x = 0–1.0, y = 0.20, z = 0.30, respectively), have been designed for investigating the variation of the intermediate O‐phase region and its effect on the electrical properties. The temperature‐composition phase diagram of each group has been proposed based on the X‐ray diffraction patterns and the temperature‐dependent dielectric behaviors. The enhanced piezoelectric properties are achieved in the O–T phase boundary compositions of the three groups, which are BH0.16T–0.58BC0.20T, BH0.16T–0.48BC0.30T and BH0.20T–0.53BC0.30T, respectively. Piezoelectric coefficient d33 of 448 pC/N is obtained in BH0.20T–0.53BC0.30T, which is higher than those of the other two phase boundary compositions. The O‐phase zone in BH0.20T–xBC0.30T is narrower than those in the BH0.16T–xBC0.20T and BH0.16T–xBC0.30T. In spite of its small O‐phase volume occupancy, the O‐phase plays a key role in the properties of the system. Our work confirms that the enhancement in piezoelectric properties is not only related to the O–T phase boundary near room temperature (RT), but also related to the shift of TR‐O toward RT. In addition, a quantitative relation between the phase boundary composition and atomic mole ratio of Hf to Ca (Rm) is proposed, which the Rm value corresponding to the phase boundary is about 0.58. The results clearly demonstrate that in this system high piezoelectric properties are achieved by tuning the specific Rm value. Such a work may provide new clues for designing lead‐free piezoelectric materials with enhanced piezoelectric property.