The COVID-19 pandemic and the accompanying circumstances (lockdown and social distancing) have been found to cause lifestyle habit changes. While negative changes (adopting risky behaviors) are known to be associated with high psychological distress, the effect of positive changes (adopting health-enhancing behaviors) has yet to be investigated. This study examined the association between the nature of changes and psychological distress, in addition to the moderating effect of “COVID-related stressors”. Online questionnaires, completed by 1969 participants, assessed the following: lifestyle changes due to COVID-19 with regard to cigarette smoking, alcohol consumption, physical activity, sleep quality, and eating habits; COVID-related stressors; Depression, Anxiety, and Stress Scale. Positive and negative changes were associated with higher psychological distress compared to no lifestyle change, and there was a moderating effect of COVID-related stressors in the association between COVID-related lifestyle changes and distress. In conclusion, to reduce psychological distress in overwhelming circumstances such as the COVID-19 pandemic, people should carefully consider whether to make changes in their lifestyle habits before doing so, even if these changes are perceived as positive and health-enhancing. Furthermore, efforts should be focused on reducing the perception of stress from COVID-19 by working on personal and mental perceptions of the situation.