Inferring optical response from other correlated optical response is highly demanded for vast applications such as biological imaging, material analysis, and optical characterization. This is distinguished from widely-studied forward and inverse designs, as it is boiled down to another different category, namely, spectra-to-spectra design. Whereas forward and inverse designs have been substantially explored across various physical scenarios, the spectra-to-spectra design remains elusive and challenging as it involves intractable many-to-many correspondences. Here, we first dabble in this uncharted area and propose a generation-elimination framework that can self-orient to the best output candidate. Such a framework has a strong built-in stochastically sampling capability that automatically generate diverse nominations and eliminate inferior nominations. As an example, we study terahertz metasurfaces to correlate the reflection spectra from low to high frequencies, where the inaccessible spectra are precisely forecasted without consulting structural information, reaching an accuracy of 98.77%. Moreover, an innovative dimensionality reduction approach is executed to visualize the distribution of the abstract correlated spectra data encoded in latent spaces. These results provide explicable perspectives for deep learning to parse complex physical processes, rather than “brute-force” black box, and facilitate versatile applications involving cross-wavelength information correlation.