We present a novel ''spectral subtraction algorithm'' (SSA) technique to remove speckle noise. It consists of a loworder and a high-order SSA and is based on a three-dimensional image spectroscopy in which the three-dimensional data cube is available and thus the speckle noise introduced by the wave-front error can be efficiently subtracted. For the low-order SSA, speckles up to the second or third order can be totally subtracted, leaving the residual speckles dominated only by the third or fourth order, respectively, and imaging contrast is increased consequently; for the high-order SSA, speckles up to the fourth or fifth order can be subtracted, leaving the residual speckles dominated only by the fifth or sixth order, respectively, and the performance is further improved. This is the first demonstration that such high-order speckles could be subtracted. Since the SSAs are conducted over a wide spectral band, a whitelight image can be re-assembled from the three-dimensional data cube. The white-light image would increase the single-to-noise ratio and reduce the exposure time, which are crucial for the search of faint companion objects. Combined with a coronagraph, the SSA can provide an extra contrast gain for the coronagraph imaging, relax the requirement for the wave-front quality (no adaptive optics correction is required for a space-borne imaging system), and significantly increase the performance of exoplanet imaging and biomarker spectroscopy.