Low-frequency noise emitted by elevated viaducts in light rail transit lines has considerable adverse effects on inhabitants living nearby. Reliable prediction of the acoustic field radiating from the viaduct using forward numerical models is challenging because the model parameters that influence viaduct vibration are difficult to obtain. To avoid directly quantifying these parameters, such as wheel–rail combined roughness, an inverse method is presented to reconstruct the acoustic field using modal acoustic transfer vectors and the sound pressures at a small number of measurement points. First, a forward numerical method based on the train–track–bridge interaction analysis is performed to predict the structure-borne noise of a concrete box girder viaduct. Then, the calculated sound pressures are treated as virtual measurement results to illustrate the inverse method procedure. Both QR decomposition and singular value decomposition with Tikhonov regularisation are used in the inverse analysis. Third, the proposed inverse method is validated by comparing the sound pressure levels computed using the inverse method with the results simulated using the forward prediction method. Finally, the reliability of the inverse procedure is further validated through field tests of two U-shaped girder viaducts.