Background: There is still a need for studies on the quality of life (QoL) at work among COVID-19 survivors. Therefore, we aimed to evaluate the association between the brain fog symptoms and the QoL at work in non-hospitalized patients with previous SARS-CoV-2 infection. Methods: Three hundred non-hospitalized patients (79.33% women; median age, 36 years; interquartile range, 30–48 years) were included in the final analysis. An anonymous neuropsychological questionnaire containing eight different questions on the presence of brain fog symptoms in four time intervals, i.e., pre-COVID-19 and 0–4, 4–12, and >12 weeks after infection, was retrospectively introduced to patients and staff of the University Hospital in Krakow. Additionally, a four-point Likert scale was used to evaluate QoL at work in four time periods. Included were participants aged ≥18 years in whom the diagnosis of COVID-19 was confirmed by the RT-PCR from nasopharyngeal swab and the first symptoms occurred no earlier than 3 months before the completion of the questionnaire. Results: Before SARS-CoV-2 infection, 28.00% (n = 84) of patients reported poor QoL at work. Within 4, 4–12, and >12 weeks after infection, a decrease in QoL was observed in 75.67% (n = 227), 65.00% (n = 195), and 53.66% (n = 161) of patients, respectively (p < 0.001). With increasing deterioration of the QoL at work, the number of brain fog symptoms increased, and patients with severe QoL impairment exhibited a median of five symptoms for <4, 4–12, and >12 weeks post-COVID-19. In the multivariable logistic regression model, predictors of the deterioration of the QoL at work depended on the time from COVID-19 onset; in the acute phase of the disease (<4 weeks), it was predicted by impairment in remembering information from the past (OR 1.88, 95%CI: 1.18–3.00, p = 0.008) and multitasking (OR 1.96, 95%CI: 1.48–2.58, p < 0.001). Furthermore, an impairment in the QoL at work 4–12 weeks and >12 weeks after COVID-19 was independently associated with age (OR 0.46, 95%CI: 0.25–0.85, p = 0.014 and OR 1.03, 95%CI: 1.01–1.05, p = 0.025, respectively), problems with multitasking (OR 2.05, 95%CI: 1.40–3.01, p < 0.001 and OR 1.75, 95%CI: 1.15–2.66, p = 0.009, respectively), answering questions in an understandable/unambiguous manner (OR 1.99, 95%CI: 1.27–3.14, p = 0.003 and OR 2.00, 95%CI: 1.47–2.36, p = 0.001, respectively), and, only for the >12 week interval, problems with remembering information from the past (OR 2.21, 95%CI: 1.24–3.92, p = 0.007). Conclusions: Certain brain fog symptoms, such as impaired memory or multitasking, are predictors of a poorer QoL at work not only during the acute phase of COVID-19 but also within more than 12 weeks after the onset of infection.