A signal-processing method is proposed in the fully interferometric three-dimensional (3D) imaging spectrometry. This processing computes a 3D interferogram, in which recorded fringe patterns do not directly reflect wavefront forms propagated from a polychromatic light source under measurement. This paper presents a procedure for signal processing including a synthesis of the 3D interferogram and retrieval of a set of spectral components of 3D images. We demonstrate retrieving 3D images for spectral components of two planar light sources by means of the proposed method. The procedure to synthesize the 3D interferogram in this method suggests the possibility of direct measurement of the 3D interferogram.