Over the past decade, the tumor microenvironment (TME) has become a new paradigm of cancer diagnosis and therapy due to its unique biological features, mainly the interconnection between cancer and stromal cells. Within the TME, cancer-associated fibroblasts (CAFs) demonstrate as one of the most critical stromal cells that regulate tumor cell growth, progression, immunosuppression, and metastasis. CAFs are identified by various biomarkers that are expressed on their surfaces, such as fibroblast activation protein (FAP), which could be utilized as a useful target for diagnostic imaging and treatment. One of the advantages of targeting FAP-expressing CAFs is the absence of FAP expression in quiescent fibroblasts, leading to a controlled targetability of diagnostic and therapeutic compounds to the malignant tumor stromal area using radiolabeled FAP-based ligands. FAP-based radiopharmaceuticals have been investigated strenuously for the visualization of malignancies and delivery of theranostic radiopharmaceuticals to the TME. This review provides an overview of the state of the art in TME compositions, particularly CAFs and FAP, and their roles in cancer biology. Moreover, relevant reports on radiolabeled FAP inhibitors until the year 2021 are highlighted—as well as the current limitations, challenges, and requirements for those radiolabeled FAP inhibitors in clinical translation.