The emergence of Zika-virus-associated congenital microcephaly has engendered renewed interest in the pathogenesis of microcephaly induced by infectious agents. Three of the original "TORCH" agents are associated with an appreciable incidence of congenital microcephaly: cytomegalovirus, rubella virus, and Toxoplasma gondii. The pathology of congenital microcephaly is characterized by neurotropic infectious agents that involve the fetal nervous system, leading to brain destruction with calcifications, microcephaly, sensorineural hearing loss, and ophthalmologic abnormalities. The inflammatory reaction induced by these four agents has an important role in pathogenesis. The potential role of "strain differences" in pathogenesis of microcephaly by these four pathogens is examined. Specific epidemiologic factors, such as first and early second trimester maternal infection, and the manifestations of congenital infection in the infant, shed some light on the pathogenesis. Immune aspects of normal pregnancy and their role in congenital infections is examined. In this review, we integrate all these findings to create a unified hypothesis of the pathogenesis of congenital microcephaly induced by these infectious agents.