The brain criticality hypothesis postulates that brain dynamics are set at a phase transition where information processing is optimized. Long-range temporal correlations (TCs) characterizing the dissipation of information within a signal have been shown to be a hallmark of brain criticality. However, the experimental link between cognitive performance, criticality and thus TCs has remained elusive due to limitations in recording length, and spatial and temporal resolution. In this study, we investigate multi-day invasive EEG recordings of 104 persons with epilepsy together with an extensive cognitive test battery. We show that short TCs predict cognitive impairment. Further, we show that heterogeneous factors, including interictal epileptiform discharges (IEDs), antiseizure medications (ASMs) and intermittent periods with slow-wave activity (SWSs), all act directly to perturb critical dynamics and thus cognition. Our work suggests critical dynamics to be the setpoint to measure optimal network function, thereby providing a unifying framework for the heterogeneous mechanisms impacting cognition in conditions like epilepsy.