Background: In oncology, lesion characterization is essential for tumor grading, treatment planning, and follow-up of cancer patients. Hybrid imaging systems, such as Single Photon Emission Computed Tomography (SPECT)/CT, Positron Emission Tomography (PET)/CT, or PET/magnetic resonance imaging (MRI), play an essential role for the noninvasive quantification of tumor characteristics. However, most of the existing approaches are challenged by intra-and intertumor heterogeneity. Novel quantitative imaging parameters that can be derived from textural feature analysis (as part of radiomics) are promising complements for improved characterization of tumor heterogeneity, thus, supporting clinically relevant implementations of personalized medicine concepts. Nevertheless, establishing new quantitative parameters for tumor characterization requires the use of standardized imaging objects to test the reliability of results prior to their implementation in patient studies. Methods: In this review, we summarize existing reports on heterogeneous phantoms with a focus on simulating tumor heterogeneity. We discuss the techniques, materials, advantages, and limitations of the existing phantoms for PET, CT, and MR imaging modalities. Conclusions: Finally, we outline the future directions and requirements for the design of cross modality imaging phantoms.