Ultrathin bismuth (Bi) layers on Si(111)-7 × 7 undergo a structural phase transformation with reducing the number of atomic layers at 3 bilayers (BL). We investigate the phonon dynamics of the Bi films close to the phase transformation by pump-probe reflectivity measurements. Coherent A 1g and E g phonons at 3 and 2 THz are clearly observed for the Bi layers with thicknesses down to 3 BL, confirming their rhombohedral crystalline structure. The A 1g frequency exhibits an abrupt redshift and splits into two components at 3 BL, which are attributed to the vertical motions of Bi atoms localized at the surface and subsurface bilayers. The E g frequency, by contrast, shows a gradual blueshift with reducing the thickness, possibly due to the lateral compressive stress at the Bi/Si interface. Below 3 BL, no coherent phonon signal is detected, in agreement with the phase transformation to the black-phosphoruslike structure. Our observations indicate that the vertical vibrations are significantly softened at 3 BL, but become almost as hard as those in the bulk crystal by adding another bilayer.