An accurate single‐crystal X‐ray diffraction study of bismuth‐containing HoFe3(BO3)4 between 11 and 500 K has revealed structural phase transition at Tstr = 365 K. The Bi atoms enter the composition from Bi2Mo3O12‐based flux during crystal growth and significantly affect Tstr. The content of Bi was estimated by two independent methods, establishing the composition as (Ho0.96Bi0.04)Fe3(BO3)4. In the low‐temperature (LT) phase below Tstr the (Ho0.96Bi0.04)Fe3(BO3)4 crystal symmetry is trigonal, of space group P3121, whereas at high temperature (HT) above 365 K the symmetry increases to space group R32. There is a sharp jump of oxygen O1 (LT) and O2 (LT) atomic displacement parameters (ADP) at Tstr. O1 and O2 ADP ellipsoids are the most elongated over 90–500 K. In space group R32 specific distances decrease steadily or do not change with decreasing temperature. In space group P3121 the distortion of the polyhedra Ho(Bi)O6, Fe1O6 and Fe2O6, B2O3 and B3O3 increases with decreasing temperature, whereas the triangles B1O3 remain almost equilateral. All BO3 triangles deviate from the ab plane with decreasing temperature. Fe–Fe distances in Fe1 chains decrease, while distances in Fe2 chains increase with decreasing temperature. The Mössbauer study confirms that the FeO6 octahedra undergo complex dynamic distortions. However, all observed distortions are rather small, and the general change in symmetry during the structural phase transition has very little influence on the local environment of iron in oxygen octahedra. The Mössbauer spectra do not distinguish two structurally different Fe1 and Fe2 positions in the LT phase. The characteristic temperatures of cation thermal vibrations were calculated using X‐ray diffraction and Mössbauer data.