Context. In the solar atmosphere, the acoustic cutoff frequency is a local quantity which depends on the atmospheric height. It separates the low-frequency evanescent waves from the high-frequency propagating waves. Aims. We measure the cutoff frequency of slow magnetoacoustic waves at various heights of a sunspot umbra and compare the results with the estimations from several analytical formulae. Methods. We analyzed the oscillations in the umbra of a sunspot belonging to active region NOAA 12662 observed in the 10830 Å spectral region with the GREGOR Infrared Spectrograph and in the Fe i 5435 Å line with the GREGOR Fabry-Pérot Interferometer. Both instrumets are attached to the GREGOR telescope at the Observatorio del Teide, Tenerife, Spain. We have computed the phase and amplification spectra between the velocity measured from different pairs of lines that sample various heights of the solar atmosphere. The cutoff frequency and its height variation have been estimated from the inspection of the spectra. Results. At the deep umbral photosphere the cutoff frequency is around 5 mHz and it increases to 6 mHz at higher photospheric layers. At the chromosphere the cutoff is ∼ 3.1 mHz. The comparison of the observationally determined cutoff with the theoretically predicted values reveals an agreement in the general trend and a reasonable match at the chromosphere, but also significant quantitative differences at the photosphere. Conclusions. Our analyses show strong evidence of the variation of the cutoff frequency with height in a sunspot umbra, which is not fully accounted for by current analytical estimations. This result has implications for our understanding of wave propagation, the seismology of active regions, and the evaluation of heating mechanisms based on compressible waves.