Humans adapt their life history strategy and lifestyle behavior to the environment. The social restrictions during the COVID-19 pandemic in the years 2020–2021 led to changes in diet and physical activity habits in large parts of populations worldwide. Additionally, many people reported an increase in body weight during the pandemic. We investigated how diet quality, physical activity, and BMI changed since the end of the COVID-19 restrictions in Switzerland (2019–2023). We assessed diet and physical activity using a standardized questionnaire. We measured anthropometry and BMI using a stadiometer and a balance of a medical bioimpedance analyzer. Our results show a stable BMI in men and women during the pandemic. In both sexes, diet quality and physical activity levels did not change significantly during this time. Among women, physical activity and age were associated with BMI, while in men, apart from physical activity, high diet quality and living in the agglomeration were associated with BMI. Overall, we did not find evidence for a transient “unhealthier” lifestyle immediately after the COVID-19 pandemic. However, we found constituent sex differences in BMI determinants. Our results could indicate a higher resilience in highly educated persons and might contribute to more personalized approaches to prevent obesity.