Short- and long-term neurodevelopmental (ND) disabilities with negative impact on psychosocial and academic performance, quality of life, and independence in adulthood are known to be the most common sequelae for surviving children after surgery for congenital heart disease (CHD). This article reviews influences and risk factors for ND impairment. For a long time, the search for independent risk factors was focused on the perioperative period and modalities of cardiopulmonary bypass (CPB). CPB operations to ensure intraoperative vital organ perfusion and oxygen supply with or without circulatory arrest or regional cerebral perfusion bear specific risks. Examples of such risks are embolization, deep hypothermia, flow rate, hemodilution, blood gas management, postoperative hyperthermia, systemic inflammatory response, and capillary leak syndrome. However, influences of these procedure-specific risk factors on ND outcome have not been found as strong as expected. Furthermore, modifications have not been found to support the effectiveness of the currently used neuroprotective strategies. Postoperative factors, such as need for extracorporal membrane oxygenation or assist device support and duration of hospital stay, significantly influence ND parameters. On the other hand, the so-called “innate,” less modifiable patient-specific risk factors have been found to exert significant influences on ND outcomes. Examples are type and severity of CHD, genetic or syndromic abnormalities, as well as prematurity and low birth weight. Structural and hemodynamic characteristics of different CHDs are assumed to result in impaired brain growth and delayed maturation with respect to the white matter. Beginning in the fetal period, this so-called “encephalopathy of CHD” is suggested a major innate risk factor for pre-, peri-, and postoperative additional hypoxic or ischemic brain injury and subsequent ND impairment. Furthermore, MRI studies on brain volume, structure, and function in adolescents have been found correlated with cognitive, motor, and executive dysfunctions. Finally, family and environmental factors independently moderate against ND outcomes. In conclusion, the different mediating factors may exert independent effects on ND and interactive influences. Implications for the future comprise modifying clinical risk factors, such as perioperative cerebral oxygen delivery, conducting brain MRI studies in correlation to ND outcomes, and extending psychosocial interventions leading to adequate resilience.