Many breast and prostate cancers are driven by the action of steroid hormones on their cognate receptors in primary tumors and in metastases, and endocrine therapies that inhibit hormone production or block the action of these receptors provide clinical benefit to many but not all of these cancer patients. Because it is difficult to predict which individuals will be helped by endocrine therapies and which will not, positron emission tomography (PET) imaging of estrogen receptor (ER) and progesterone receptor (PgR) in breast cancer, and androgen receptor (AR) in prostate cancer can provide useful, often functional, information on the likelihood of endocrine therapy response in individual patients. This review covers our development of three PET imaging agents, 16α-[18F]fluoroestradiol (FES) for ER, 21-[18F]fluoro-furanyl-nor-progesterone (FFNP) for PgR, and 16β-[18F]fluoro-5α-dihydrotestosterone (FDHT) for AR, and the evolution of their clinical use. For these agents, the pathway from concept through development tracks with an emerging understanding of critical performance criteria that is needed for successful PET imaging of these low-abundance receptor targets. Progress in the ongoing evaluation of what they can add to the clinical management of breast and prostate cancers reflects our increased understanding of these diseases and of optimal strategies for predicting the success of clinical endocrine therapies.