Many previous studies have indicated the adverse effects of bisphenol A (BPA) on sperm production and quality; however, the mechanisms underlying BPA male reproductive toxicity have yet to be elucidated. The main purpose of this study was to investigate the effect of perinatal exposure to BPA on the spermatogenic capacity of male offspring, and to explore the possible influence of inflammatory responses in BPA reproductive toxicity. Twenty-one pregnant C57BL/6mice were randomly divided into three groups: a control group, a group receiving 0.2 μg/mL (LBPA), and a group receiving 2 μg/mL of BPA (HBPA), all via drinking water from gestational day 6 to the end of lactation. After weaning, one male mouse was randomly selected from each group (n = 7/group); these three mice were fed a normal diet and drinking water for 1 month. Levels of serum testosterone (T) and tumor necrosis factor (TNF)-α were then measured in all mice. Sperm count and the proportion of sperm malformation were also determined. The levels of Toll-like receptor 4 (TLR4), nuclear factor (NF)-κB, and aryl hydrocarbon receptor (AhR) protein expression in the testis tissue were determined. Analysis showed that the proportion of sperm malformation increased in the LBPA and HBPA groups (p < 0.05). Sperm count significantly decreased only in the HBPA group (p < 0.05), while the levels of serum TNF-α increased in the LBPA and HBPA groups (p < 0.05). Levels of serum T decreased significantly in the HBPA group, compared with controls (p < 0.05). Levels of TLR4 and NF-κB protein expression in the testis were significantly higher in the LBPA and HBPA groups (p < 0.05 or p < 0.01), while AhR protein expression was higher and seminiferous tubules in the testis showed more damage in the HBPA group compared to controls (p < 0.05 and p < 0.01, respectively). Our results showed that perinatal exposure to low or high doses of BPA decreased the capacity for spermatogenesis in male offspring, which may be associated with an inflammatory response activated by the TLR4/ NF-κB and AhR signaling pathways in the testis.