Androgen production, being important for male fertility, is mainly accomplished by the Leydig cells from the interstitial compartment of the testis. Testosterone plays a critical role in testis development, normal masculinization, and the maintenance of spermatogenesis. Within seminiferous tubules, appropriate Sertoli cell function is highly dependent on testicular androgen levels and is essential to initiate and maintain spermatogenesis. During aging, testosterone production by the testicular Leydig cells declines from the 30s in humans at a rate of 1% per year. This review outlines the recent findings regarding the use of flavonoids and isoflavonoids to improve testosterone production, contributing to normal spermatogenesis and preventing age-related degenerative diseases associated with testosterone deficiency. With the cumulation of information on the actions of different flavonoids and isoflavonoids on steroidogenesis in Leydig cells, we can now draw conclusions regarding the structure-activity relationship on androgen production. Indeed, flavonoids having a 5,7-dihydroxychromen-4-one backbone tend to increase the expression of the steroidogenic acute regulatory protein (StAR), being critical for the entry of cholesterol into the mitochondria, leading to increased testosterone production from testis Leydig cells. Therefore, flavonoids and isoflavonoids such as chrysin, apigenin, luteolin, quercetin, and daidzein may be effective in delaying the initiation of late-onset hypogonadism associated with aging in males.