The phonon spectra and structural characteristics of the LuFe 2 O 4 system were investigated using X-ray diffraction as well as Raman and infrared (IR) spectroscopic techniques. Two step-like anomalies of the wavenumbers of the E 3 g peak in the Raman spectra and the E 3 u peak in IR spectra, as well as some weak anomalies of the lattice parameters, were found around the ferrimagnetic and ferroelectric transition temperatures. The substitutions of Lu with Yb and Fe with Mn lead to the corresponding wavenumber shifts in the optical spectra, which can be attributed to the changes in the atomic mass and lattice parameters. Furthermore, the Raman intensity was found to be related to the changes in the local structure and electrical conductivity. These results illustrate that the structural fluctuations could effectively influence the phonon modes, and a possible interplay among the structural, magnetic, and charge-ordering properties exists in this multiferroic system. Copyright