In this paper, multiple interference components in a fiber Mach–Zehnder interferometer sensor and their influence on wavelength shift tracing were studied, based on which a frequency filtering method followed by inverse Fourier transform was proposed to improve the sensor’s response accuracy. Firstly, multiple interference components in the transmission spectrum of a fiber MMF-SMF-MMF structure were simulated, and environmental response discrepancy in different components was calculated. Then, Fourier transform was performed on the transmission spectrum followed by frequency filtering to pick out the dominant interference component. An inverse Fourier transform was then carried out to obtain the wavelength shift which was undisturbed by minor interference components. The method was demonstrated by temperature and ambient refractive index (RI) calibration experiments as well as verification tests. The deviation between temperature sensitivity obtained with and without frequency filtering was 39%. The verification test at 80 °C was carried out and the measurement error was three times lower due to frequency filtering. For RI sensing, the deviation between sensitivity obtained with and without filtering was 19%. Measurement errors at 1.3706 of RI were seven times decreased after frequency filtering. Due to its simplicity and efficiency, the proposed method is believed to have extensive application prospects in sensing fields, especially when high-precision temperature and RI sensing are desired.