Patients recovering from COVID-19 commonly exhibit cognitive and brain alterations, yet the specific neuropathological mechanisms and risk factors that underlie these alterations remain elusive. Given the significant global incidence of COVID-19, identifying factors that can distinguish individuals at risk of developing medium or long-term brain alterations is crucial for prioritizing follow-up care. Here, we report findings from a sample of 100 patients who were affected by a respiratory infection during the COVID-19 pandemic. This sample comprised 73 adults with a mild to moderate SARS-CoV-2 infection (who did not require invasive ventilatory assistance) and 27 with infections attributed to other agents and no history of COVID-19. The participants underwent cognitive screening, a decision-making task to measure cognitive flexibility, and magnetic resonance imaging evaluations. We assessed two clinical factors during infection: the presence of anosmia and the requirement for hospitalization due to respiratory symptoms. Groups did not differ in age or cognitive performance, but clinical factors differentially affected task performance. Patients who presented anosmia during the acute episode exhibited more impulsive changes in alternatives after a shift in probabilities in the decision-making task, while patients who required hospitalization showed more perseverative choices. Interestingly, the presence of anosmia during SARS-CoV-2 infection correlated with several brain measures, including decreases in functional activity during the decision-making task, thinning of cortical thickness in parietal regions, and loss in white matter integrity in corticospinal tracts and parietal-thalamic fasciculi, among others. These results suggest that anosmia could be a risk factor for developing brain alterations after SARS-CoV-2 infection and may serve to identify at-risk populations for follow-up.