Reproductive aging in males is characterized by a diminution in sexual behavior beginning in middle age. We investigated the relationships among testosterone, androgen receptor (AR) and estrogen receptor alpha (ERα) cell numbers in the hypothalamus, and their relationship to sexual performance in male rats. Young (3 months) and middle-aged (12 months) rats were given sexual behavior tests, then castrated and implanted with vehicle or testosterone capsules. Rats were tested again for sexual behavior. Numbers of AR and ERα immunoreactive cells were counted in the anteroventral periventricular nucleus and the medial preoptic nucleus, and serum hormones were measured. Middle-aged intact rats had significant impairments of all sexual behavior measures compared to young males. After castration and testosterone implantation, sexual behaviors in middle-aged males were largely comparable to those in the young males. In the hypothalamus, AR cell density was significantly (5-fold) higher, and ERα cell density significantly (6-fold) lower, in testosterone- than vehicle-treated males, with no age differences. Thus, restoration of serum testosterone to comparable levels in young and middle-aged rats resulted in similar preoptic AR and ERα cell density concomitant with a reinstatement of most behaviors. These data suggest that age-related differences in sexual behavior cannot be due to absolute levels of testosterone, and further, the middle-aged brain retains the capacity to respond to exogenous testosterone with changes in hypothalamic AR and ERα expression. Our finding that testosterone replacement in aging males has profound effects on hypothalamic receptors and behavior has potential medical implications for the treatment of age-related hypogonadism in men.