Polarized Raman scattering and infrared reflection spectra of hexagonal HoMnO3 single crystals in the temperature range 10-300 K are reported. Group-theoretical analysis is performed and scattering selection rules for the second order scattering processes are presented. Based on the results of lattice dynamics calculations, performed within the shell model, the observed lines in the spectra are assigned to definite lattice vibrations. The magnetic ordering of Mn ions, which occurs below TN =76 K, is shown to effect both Raman-and infrared-active phonons, which modulate Mn-O-Mn bonds and, consequently, Mn exchange interaction.