Viscoelastic material can significantly reduce the vibration energy and radiated noise of a structure, so it is widely used in lightweight sandwich structures. The accurate and efficient determination of the frequency-dependent complex modulus of viscoelastic material is the basis for the correct analysis of the vibro-acoustic behavior of sandwich structures. Based on the behavior of a sandwich beam whose core is a viscoelastic layer, a combined theoretical and experimental study is proposed to characterize the properties of the viscoelastic layer constituting the core. In this method, the viscoelastic layer is bonded between two constraining layers. Then, a genetic algorithm is used to fit the analytical solution of the frequency¬ response function of the free–free constrained beam to the measured result, and then the frequency-dependent complex modulus is estimated for the viscoelastic layer. Moreover, by varying the length of the beams, it is possible to characterize the frequency-dependent complex modulus of the viscoelastic material over a wide frequency range. Finally, the characterized frequency-dependent complex modulus is imported into a finite element model to compute the complex natural frequencies of a sandwich beam, and a comparison of the simulated and measured results displays that the errors in the real parts are within 2.33% and the errors in the imaginary parts are within 3.31%. It is confirmed that the proposed method is feasible, accurate, and reliable. This provides essential technical support for improving the acoustic vibration characteristics of sandwich panels by introducing viscoelastic materials.