Inorganic metal halide perovskites are promising materials for next-generation technologies due to plethora of unique physical properties, many of which cannot be observed in the oxide perovskites. On the other hand, the search for ferroelectricity and multiferroicity in lead-free inorganic halide perovskites remains a challenging research topic. Here, we experimentally show that cubic fluoroperovskites exhibit proximity to incipient ferroelectrics, which manifested in the softening of the low-frequency polar phonons in the Brillouin zone center at cooling. Furthermore, we reveal the coupling between harmonic and anharmonic force constants of the softening phonons and their correlation with the perovskite tolerance factor. Next, using first-principles calculations, we examine the lattice dynamics of the cubic fluoroperovskites and disclose the incipient lattice instability at which the harmonic force constants of low-lying phonons tend to decrease with a reduction of tolerance factor at all high-symmetry points of the Brillouin zone. The correlations with the tolerance factor indicate the geometric origin of observed incipient lattice instability in the cubic fluoroperovskites caused by the steric effect due to the volume filling of the unit cell by different radius ions. These results provide insights into the lattice dynamics and potential ferroelectric properties of inorganic lead-free metal halide perovskites, relevant to further design and synthesis of new multifunctional materials.