During early development, continuous exposure to environmental contaminants such as bisphenol A (BPA) is known to alter neuronal development, resulting in aberrant brain structure and predisposing individuals to developing neuropsychiatric disorders later in life. While the altered oligodendrocyte (OL) structure and function have been casually linked to the occurrence of numerous psychiatric diseases, it remains open whether early BPA exposure (EBE) also recruits OLs to mediate its toxicity in the brain. Here, we observed that EBE from birth to postnatal day 21 caused a substantial loss of hippocampal OLs in rat pups. The OL loss was enduring and manifested even when the affected pups spanned into their adulthood. In parallel, the expression of two key proteins in mature OLs, myelin basic protein (MBP), and monocarboxylate transporter 1 (MCT1) was markedly downregulated in adult hippocampus with a considerable reduction in the number of myelinated axons. By contrast, the myelination of individual axons remained intact. The altered hippocampal OLs were related to EBE-mediated disruption of estrogen receptor (ER) signaling in developing OLs and could be readily prevented by treatment with low level of ICI 182780, an ER antagonist. Importantly, the adult rats subject to EBE exhibited clear deficit in contextual fear memory, which highly correlated with OL loss and decreased MBP and MCT1 expression in hippocampus. The OL loss may thus represent an alternative route through which EBE has its adversity on the brain and contributes to the development of neuropsychiatric illness.