Achieving systemic tumor control across metastases is vital for long-term patient survival but remains intractable in many patients. High lesion-level response heterogeneity persists, conferring many dissociated responses across metastatic lesions. Most studies of metastatic disease focus on tumor molecular and cellular features, which are crucial to elucidating the mechanisms underlying lesion-level variability. However, our understanding of lesion-specific heterogeneity on the macroscopic level, such as lesion dynamics in growth, response, and progression during treatment, remains rudimentary. This study investigates lesion-specific response heterogeneity through analyzing 116,542 observations of 40,612 lesions in 4,308 metastatic colorectal cancer (mCRC) patients. Despite significant differences in their response and progression dynamics, metastatic lesions converge on four phenotypes that vary with anatomical site. Importantly, we find that organ-level progression sequence is closely associated with patient long-term survival, and that patients with the first lesion progression in the liver often have worse survival. In conclusion, our study provides insights into lesion-specific response and progression heterogeneity in mCRC and creates impetus for metastasis-specific therapeutics.