During the outbreak of the COVID-19 epidemic, fear of disease and its consequences, recommended lifestyle changes, and severe restrictions set by governments acted as stressors and affected people’s mood, emotions, mental health, and wellbeing. Many studies conducted during this crisis focused on affective and physiological responses to stress, but few studies examined how the crisis affected cognition. The present cross-sectional study examined the relationship between physiological, affective, and cognitive responses to the epidemic. In an online survey conducted at the height of the first wave of the epidemic in Slovenia (April 15–25, 2020), 830 Slovenian residents aged 18–85 years reported the effects of stressors (confinement, problems at home, problems at work, lack of necessities, and increased workload), experienced emotions, generalized anxiety, perceived stress, changes in health, fatigue and sleep quality, and perceived changes in cognition during the epidemic. Risk factors for stress (neuroticism, vulnerability, general health, gender, and age) were also recorded. We hypothesized that stressors and stress risk factors will be related to subjective cognitive decline, with negative emotions, generalized anxiety, perceived stress, and physical symptoms acting as mediator variables. On average, the results showed a mild subjective cognitive decline during the epidemic. In structural equation modeling, 34% of its variance was predicted by the mediator variables, with negative emotions and physical symptoms having the largest contribution. Stress risk factors were predictably related to the four mediator variables. Among the stressors, confinement showed the strongest effect on the four mediator variables, implying the importance of thoughtful communication about necessary restrictive measures during emergency circumstances. The results of this study indicate that the possibility of altered cognitive function should be considered when planning work and study activities during the epidemic.