Stressful events and chronic tension are considered a burden and a threat to physical, mental, and social health. The aim of the study was to demonstrate the associations of variation in stress exposure with social factors, physical activity, basic components of physical fitness, body mass index (BMI) and percentage of body fat (BFP). An additional objective was to identify the main BFP modifiers among those analyzed. The material consisted of data of ethnically homogeneous group 355 men (32–87 yrs), invited to the study as part of the Wroclaw Male Study research project. The analyzed features included socioeconomic status (age, educational level), elements of lifestyle (physical activity), major and most important stressful life events—Social Readjustment Rating Scale (SRRS) and basic parameters of the somatic structure of the body (BMI, BFP). Statistical analyses included: chi-square test, Mann–Whitney U test and backward stepwise regression (significance level α = 0.05). Stress exposure showed significant socioeconomic variation among the adult Poles studied. Higher levels of education were associated with higher levels of stress. Significant correlations between SRRS and physical activity were found, especially in men older than 60 years and with higher levels of education. A positive relationship was shown between SRRS and BFP, especially in men under 60 years of age. BFP appeared to depend mainly on age and stress. The main determinants of SRRS were age and education level, while BFP turned out to be more sensitive to stress than BMI. The modifying force of physical activity for SRRS appears to be age dependent.