As a “marine ecological engineer”, the oyster reefs not only perform important ecological functions, but also reduce the damage caused by waves to protective structures such as seawalls. However, oyster reefs in shallow water change the nonlinear characteristics of waves and affect sediment transport and coastal evolution. Based on Fourier spectrum and analysis of Wavelet Transform, the influence of artificial bag oyster reefs on the energy and nonlinear phase coupling of irregular waves are studied through physical experiment. The results show that oyster reefs have a substantial effect on the energy of primary harmonic, which transfer to higher harmonics through triad interactions, and a considerable reduction in primary harmonic energy and an increase in higher harmonics energy are reflected in the energy spectra. The transmission spectrum behind the oyster reefs shows three peaks at primary, secondary and third harmonics. The bicoherence spectrum indicates that the peaks at secondary and third harmonics mainly result from the self-coupling of the primary harmonics and phase coupling between the primary and secondary harmonics respectively. As the water depth increases, the degree of nonlinear coupling between wave components decreases, which leads to the energy of wave components at different frequencies increases. With increasing top width, the length of the shoaling region increases, and the growth of triad nonlinear interactions are observed in wavelet-based bicoherence spectra, resulting in the spectral peak energy decreasing while the secondary harmonics energy increasing in the spectrum. Finally, the potential application of an ecological system composed by “oyster reefs + mangroves” is discussed. As the effect of water depth on wave energy is much greater than that of top width, in artificial oyster reef construction, it is recommended that keep the oyster reefs non-submerged in terms of wave dissipation. Further studies should take the dynamic growth effect of oyster reefs into account.