Micro-vibration measurement methods for spacecraft structures mainly include the use of an accelerometer, laser, and magnetohydrodynamic(MHD) measurement methods. The microvibration measurement sensor developed based on the magnetohydrodynamic measurement method has no mechanical wear between internal components, a fast high-frequency vibration response, and strong anti-interference properties. To reduce the measurement error of the racetrack magnetohydrodynamic linear motion sensor developed in the laboratory, this paper investigated the sensor error, analyzed the error source, combined the BP neural network optimized by particle swarm optimization (PSO) with variational mode decomposition
(VMD), used the VMD-PSO-BP neural network to establish the error compensation model of the racetrack magnetohydrodynamic linear motion sensor, and combined the PSO-BP neural network with wavelet threshold de-noising (WTD). The WTD-PSO-BP and RBF neural networks were used to develop the error compensation model of the racetrack magnetohydrodynamic linear motion sensor. Comparing the three models, the experimental results show that the VMD-PSO-BP model has the best compensation effect. The mean absolute error (MAE) of the output signal of the racetrack magnetohydrodynamic linear motion sensor compensated by the VMD-PSO-BP neural network model was 1–2 times lower than that before compensation, the signal-to-noise ratio was 10 times higher on average, and the correlation coefficient was more than 0.95.