Older adults are at a high risk of experiencing severe complications of influenza. Receiving a vaccination is a beneficial strategy to prevent the disease and reduce the severity of influenza illnesses. This cross-sectional questionnaire-based study aimed to evaluate the influence of sociodemographic, clinical, and mental parameters as well as other potential risk factors on refusal to vaccinate against influenza among the elderly population in Poland. Furthermore, due to the prevailing COVID-19 pandemic, we put efforts into finding any statistical correlations between the fear of COVID-19 infection in patients and their attitudes toward receiving an influenza vaccination. The study was conducted in November–December 2020 in Poland on a representative nationwide sample of 500 individuals aged > 60. Of the respondents, 62 (12.4%) and 51 (10.2%) underwent influenza vaccination in 2019 and 2020, respectively. Out of ten different factors analyzed in this study, three were significantly associated with attitudes towards influenza vaccination. Participants with net income below the national average of PLN 3000 (OR = 2.37, CI 95% [1.26–4.47]), compared to those earning more than PLN 3000, had significantly higher odds of having a negative attitude towards influenza vaccination. Furthermore, respondents with <174 cm height (OR = 2.56, CI 95% [1.51–4.33]) and those with strong fear of COVID-19 infection (OR = 1.65, CI95% [1.02–2.66]) were also more likely to refrain from influenza vaccination. We believe the identification of factors limiting the willingness to receive influenza vaccination is an effective way to help clinicians focus their efforts on educating the groups of patients with the highest odds of refusing to receive the vaccine. Moreover, it may aid the design and enforcement of national solutions or the implementation of novel legislative measures and preventive programs, increasing public confidence and promoting vaccination, especially among groups at high risk of developing this disease.