The heterogeneity of an individual patient’s tumor has been linked to treatment resistance, but quantitative biomarkers to rapidly and reproducibly evaluate heterogeneity in a clinical setting are currently lacking. Using established tools available in a CAP-accredited and CLIA-certified clinical laboratory, we quantified digital pathology features on 9,225 individual circulating tumor cells (CTCs) from 179 unique metastatic castration-resistant prostate cancer (mCRPC) patients to define phenotypically distinct cell types. Heterogeneity was quantified based on the diversity of cell types in individual patient samples using the Shannon index and associated with overall survival (OS) in the 145 specimens collected prior to initiation of second or later lines of therapy. Low CTC phenotypic heterogeneity was associated with better OS in patients treated with androgen receptor signaling inhibitors (ARSI), whereas high heterogeneity was associated with better OS in patients treated with taxane chemotherapy. Overall, the results show that quantifying CTC phenotypic heterogeneity can help inform the choice between ARSI and taxanes in mCRPC patients.