In older men, low estrogen levels are associated with poor bone microarchitecture. Data on androgens are discordant. We studied the link between baseline sex steroid levels (total 17β ‐estradiol [17βE2], total testosterone [tT], calculated bioavailable 17βE2 [bio‐17βE2], and apparent free testosterone concentration [AFTC]) and bone microarchitecture deterioration assessed prospectively in a 820 older men followed for 8 years. Bone microarchitecture was assessed by HR‐pQCT at baseline, then after 4 and 8 years. At both the skeletal sites, the bone microarchitecture deterioration rate did not correlate with serum levels of tT and 17βE2. At the distal radius, cortical area (Ct.Ar) decreased more rapidly in the lowest versus the highest AFTC quartile. At the distal tibia, cortical thickness (Ct.Th) decreased and trabecular area (Tb.Ar) increased more rapidly in the highest versus the lowest AFTC quartile. At the tibia, bone mineral content (BMC), total volumetric bone mineral density (Tt.vBMD), Ct.Th, and Ct.Ar decreased, whereas Tb.Ar increased faster in the lowest versus the highest bio‐17βE2 quartile. In men who had both AFTC and bio‐17βE2 in the lowest quartile (high‐risk group), distal radius cortical vBMD (Ct.vBMD) decreased more rapidly compared with men who had both hormones in the three upper quartiles (reference group). At the distal tibia, Tt.vBMD, Ct.Th, Ct.Ar, and Ct.vBMD decreased, whereas Tb.Ar increased more rapidly in the high‐risk group versus the reference group. In men receiving androgen deprivation therapy (ADT) for prostate cancer, BMC, Tt.vBMD, Ct.Th, Ct.Ar, and Ct.vBMD decreased, whereas Tb.Ar increased more rapidly than in men not receiving ADT at both the skeletal sites. Thus, in older men followed up prospectively, low levels of bio‐17βE2, and to a smaller extent AFTC, are associated with accelerated cortical bone deterioration. Cortical bone deterioration was strongly accelerated in men receiving ADT who had very low levels of all sex steroids. © 2019 American Society for Bone and Mineral Research.