Immune-based treatment approaches are successfully used for the treatment of patients with cancer. While such therapies can be highly effective, many patients fail to benefit. To provide optimal therapy choices and to predict treatment responses, reliable biomarkers for the assessment of immune features in patients with cancer are of significant importance. Biomarkers (BM) that enable a comprehensive and repeatable assessment of the tumor microenvironment (TME), the lymphoid system, and the dynamics induced by drug treatment can fill this gap. Medical imaging, notably positron emission tomography (PET) and magnetic resonance imaging (MRI), providing whole-body imaging BMs, might deliver such BMs. However, those imaging BMs must be well characterized as being ‘fit for purpose’ for the intended use. This review provides an overview of the key steps involved in the development of ‘fit-for-purpose’ imaging BMs applicable in drug development, with a specific focus on pharmacodynamic biomarkers for assessing the TME and its modulation by immunotherapy. The importance of the qualification of imaging BMs according to their context of use (COU) as defined by the Food and Drug Administration (FDA) and National Institutes of Health Biomarkers, EndpointS, and other Tools (BEST) glossary is highlighted. We elaborate on how an imaging BM qualification for a specific COU can be achieved.