APOE-ε4 is the strongest genetic risk factor for Alzheimer’s disease (AD), and is associated with an increase in the levels of amyloid deposition and an early age of onset. Recent data demonstrate that AD pathological changes occur decades before clinical symptoms, raising questions about the precise onset of the disease. Now a convergence of approaches in mice and humans has demonstrated that APOE-ε4 affects normal brain function even very early in life in the absence of gross AD pathological changes. Normal mice expressing APOE4 have task-specific spatial learning deficits, as well as reduced NMDAR-dependent signaling and structural changes to presynaptic and postsynaptic compartments in neurons, particularly in hippocampal regions. Young humans possessing APOE-ε4 are more adept than APOE-ε4 negative individuals at some behavioral tasks, and functional magnetic resonance imaging has shown that inheritance of APOE-ε4 has specific effects on medial temporal brain activities. These findings suggest that inheritance of APOE-ε4 causes life long changes to the brain that may be related to the late risk of AD. Several possible mechanisms of how APOE-ε4 could affect brain neurochemistry, structure, and function are reviewed.