In this paper, an asymmetric vibroacoustic system that can passively realize nonreciprocal transmission of acoustic energy is reported. This experimental system consists of a waveguide, a strongly nonlinear membrane, and three acoustic cavities with different sizes. The theoretical modeling of the system is verified by experiments, and parametric analysis is also carried out. These intensive studies reveal the nonreciprocal transmission of acoustic energy in this prototype system. Under forward excitation, internal resonance between the two nonlinear normal modes of the vibroacoustic system occurs, and acoustic energy is irreversibly transferred from the waveguide to the nonlinear membrane. However, under backward excitation, there is no internal resonance in the system. Energy spectra and wavelet analysis are used to highlight the mechanism of nonreciprocal transfer of acoustic energy. Consequently, nearly unidirectional (preferential) transmission of acoustic energy transfer is shown by this system. The nonreciprocal acoustic energy transfer method illustrated in this paper provides a new way to design the odd acoustic element.