Background: Depression and cognitive impairment are recognized complications of COVID-19. This study aimed to assess cognitive performance in clinically diagnosed post-COVID depression (PCD, n = 25) patients using neuropsychological testing. Methods: The study involved 71 post-COVID patients with matched control groups: recovered COVID-19 individuals without complications (n = 18) and individuals without prior COVID-19 history (n = 19). A post-COVID depression group (PCD, n = 25) was identified based on psychiatric diagnosis, and a comparison group (noPCD, n = 46) included participants with neurological COVID-19 complications, excluding clinical depression. Results: The PCD patients showed gender-dependent significant cognitive impairment in the MoCA, Word Memory Test (WMT), Stroop task (SCWT), and Trail Making Test (TMT) compared to the controls and noPCD patients. Men with PCD showed worse performances on the SCWT, in MoCA attention score, and on the WMT (immediate and delayed word recall), while women with PCD showed a decline in MoCA total score, an increased processing time with less errors on the TMT, and worse immediate recall. No differences between groups in Sniffin’s stick test were found. Conclusions: COVID-related direct (post-COVID symptoms) and depression-mediated (depression itself, male sex, and severity of COVID-19) predictors of decline in memory and information processing speed were identified. Our findings may help to personalize the treatment of depression, taking a patient’s gender and severity of previous COVID-19 disease into account.