Background: Cancer patients may fail to distinguish COVID-19 symptoms such as anosmia, dysgeusia/ageusia, anorexia, headache, and fatigue, which are frequent after cancer treatments. We aimed to identify symptoms associated with COVID-19 and to assess the strength of their association in cancer and cancer-free populations. Methods: The multicenter cohort study PAPESCO-19 included 878 cancer patients and 940 healthcare workers (HCWs). At baseline and quarterly thereafter, they reported the presence or absence of 13 COVID-19 symptoms observed over 3 months and the results of routine screening RT-PCR, and they were systematically tested for SARS-CoV-2-specific antibodies. We identified the symptom combinations significantly associated with COVID-19. Results: Eight percent of cancer patients were COVID-19 positive, and 32% were symptomatic. Among the HCWs, these proportions were 9.5 and 52%, respectively. Anosmia, anorexia, fever, headache, and rhinorrhea together accurately discriminated (c-statistic = 0.7027) COVID-19 cases from cancer patients. Anosmia, dysgeusia/ageusia, muscle pain, intense fatigue, headache, and chest pain better discriminated (c-statistic = 0.8830) COVID-19 cases among the HCWs. Anosmia had the strongest association in both the cancer patients (OR = 7.48, 95% CI: 2.96–18.89) and HCWs (OR = 5.71, 95% CI: 2.21–14.75). Conclusions: COVID-19 symptoms and their diagnostic performance differ in the cancer patients and HCWs. Anosmia is associated with COVID-19 in cancer patients, while dysgeusia/ageusia is not. Cancer patients deserve tailored preventive measures due to their particular COVID-19 symptom pattern.