As the pandemic continues, many older adults are facing prolonged isolation and stress while having less access to traditional ways of coping. There is widespread concern that the situation is increasingly taking its toll on older adults’ psychological and social well-being. We use linear mixed models to examine psychosocial impacts and predictors thereof among older Norwegians in early and later stages of the pandemic. Longitudinal data were collected online in the Norwegian Counties Public Health Survey right before the pandemic and in June and November–December 2020 in two counties (baseline n = 4,104; age 65–92). Outcomes include loneliness (single item, UCLA3), psychological ill-being (worried, anxious, depressed), and psychological well-being (satisfied, engaged, happy). From before to three months into the pandemic men’s psychosocial well-being remained stable, whereas women’s slightly declined. Five months later we observe broad and substantial declines in psychosocial well-being. These impacts disproportionately affect women (all outcomes) and single and older individuals (loneliness only) and are not moderated by educational level, urbanicity, or whether self or partner are reported “at risk” due to health problems. Pre-pandemic low social support and high psychological distress predict relatively improved psychosocial well-being. Older Norwegians seemed to manage the pandemic’s early stage without clear psychosocial impacts. However, we observe notably compromised well-being during the second wave of COVID-19 in late 2020. Lessons learned about the nature and distribution of the psychosocial impacts of prolonged health-threats and social distancing provide valuable knowledge for intervention design during this and future pandemics.