Extreme ultraviolet pulses as generated by high harmonic generation (HHG) are a powerful tool for both time-resolved spectroscopy and coherent diffractive imaging. However, the integration of spectroscopy and microscopy to harness the unique broadband spectra provided by HHG is hardly explored due to the challenge to decouple spectroscopic and microscopic information. Here, we present an interferometric approach to this problem that combines Fourier transform spectroscopy (FTS) with Fourier transform holography (FTH). This is made possible by the generation of phase-locked pulses using a pair of HHG sources. Crucially, in our geometry the number of interferometric measurements required is at most equal to the number of high-harmonics in the illumination, and can be further reduced by incorporating prior knowledge about the structure of the FTH sample. Compared to conventional FTS, this approach achieves over an order of magnitude increase in acquisition speed for full spectro-microscopic data, and furthermore allows high-resolution computational imaging.