Aging has been implicated in the alteration of taste acuity. Diet can affect taste sensitivity. We aimed to investigate the types of tastes altered in elderly Korean people and factors associated with taste alteration in relation to dietary intake and other factors. Elderly participants (≥65 years) and young adults were assessed to determine their recognition thresholds (RT) for sweet, salty, bitter, sour, and umami tastes. Elderly participants were further surveyed for dietary intake and non-nutritional factors. Five taste RTs were correlated with age, but only four taste RTs, except sweetness, differed between the elderly participants and young adults. Inadequate intake of iron, thiamin, folic acid, zinc, and phosphorus among the elderly participants was related to elevated taste RT levels, except for bitter taste. In both correlation and regression analyses, only salty and sour RTs were associated with energy, iron, thiamin, fiber, vitamin C, and riboflavin levels in the elderly participants. The elderly participants’ taste RTs exhibited strong associations with quality of life (QOL) but showed partial relationships with physical activity, number of medicine intakes, social gatherings, and education. Taste sensitivity may decrease with age, which is further influenced by insufficient dietary intake, especially iron and thiamin, and QOL.