Anxiety and depression in the elderly individuals have been studied around the world, and some authors consider them among the most serious problems faced by modern societies. With recent economic crisis—very important in Southern European countries—isolation, loneliness, and exclusion of the active society, mental problems are probably raising and associated with distinct factors. In this cross-sectional analysis, nested in a longitudinal population-based cohort study, we analyze anxiety and depression prevalence, and their related factors, in a representative cohort of Portuguese seniors. We used data retrieved from second wave of follow-up of EpiDoC Cohort—EpiDoC 2 study, which is composed by 10,661 adults, representative of adult Portuguese population. This study included all ≥65 years old EpiDoC 2 study participants, who responded to Hospital Anxiety and Depression Scale (HADS), n = 1,680. Sociodemographic, lifestyles, self-reported non-communicable diseases, health-related quality of life (EQ-5D-3D), physical function (HAQ), and health resources consumption data were collected. Anxiety and depression were assessed with HADS. Anxiety and depression prevalence were estimated. Multivariable logistic regression was used to assess anxiety and depression score determinants. The estimated prevalence of anxiety among Portuguese elderly is 9.6% and depression is 11.8%. Seniors with anxiety and seniors with depression have a higher probability to self-report higher levels of physical disability (OR = 3.10; 96% CI 2.12–4.52; OR = 3.08, 95% CI 2.29–4.14, respectively) and lower levels of quality of life (OR = 0.03, 95% CI 0.01–0.09; OR = 0.03, 95% CI 0.01–0.06, respectively). Female gender (OR = 2.77, 95% CI 1.53–5.00), low educational level (OR = 2.30, 95% CI 1.22–4.36), allergic (OR = 2.02, 95% CI 1.14–3.55), and rheumatic disease (OR = 2.92, 95% CI 1.74–4.90) were significantly and independently associated with the presence of anxiety symptoms. Physical inactivity (OR = 1.64, 95% CI 1.11–2.42) and low educational level (OR = 2.40, 95% CI 1.41–4.09) were significantly and independently associated with depression symptoms. Subjects that reported to drink alcohol daily or occasionally were negatively associated with depression symptoms. Anxiety and depression are frequent among Portuguese elderly. These prevalence rates suggest that preventing mental illness in senior population is a crucial need. A well-designed prevention strategy might have an effective action in raising the well-being of elderly.