Objectives: The association of body composition with survival and the efficacy of first-line treatment was investigated in patients with castration-resistant prostate cancer (CRPC). Methods: The records of CRPC patients treated with docetaxel or androgen receptor signaling inhibitors (ARSi) between 2005 and 2018 were reviewed. Skeletal muscle index (SMI), visceral fat index, and subcutaneous fat index were evaluated using pretreatment computed tomography images. Results: Of 230 eligible patients, 144 received docetaxel, and 86 received ARSi as the first-line treatment for CRPC. The SMI hi (based on median values) group had higher prostate-specific antigen (PSA) progression-free survival (median 13.5 vs. 8.3 months, p = 0.030), radiologic progression-free survival (14.9 vs. 9.1 months, p < 0.001), and overall survival (24.1 vs. 16.9 months, p = 0.015) than the SMI lo group. In docetaxel-treated patients, the SMI hi group had higher PSA progression-free survival (13.5 vs. 5.9 months, p = 0.016) and radiologic progression-free survival (14.6 vs. 6.7 months, p < 0.001) than the SMI lo group. However, PSA progression-free survival and radiologic progression-free survival were comparable between the two groups in ARSi-treated patients. SMI was independently associated with the risk of radiologic progression in patients treated with docetaxel but not in those treated with ARSi. Conclusions: High skeletal muscle mass may be associated with reduced risk of disease progression and mortality in patients with CRPC. However, the significance of these relationships is limited in patients treated with docetaxel. These results suggest that assessing skeletal muscle mass may be worthwhile when selecting treatments for CRPC; however, further prospective validation and large-scale studies are needed.