Aims/hypothesis Despite well-known sex differences in body composition it is not known whether sex-specific genetic or environmental effects contribute to these differences. Methods We assessed body composition in 2,506 individuals, from a young Dutch genetic isolate participating in the Erasmus Rucphen Family study, by dual-energy X-ray absorptiometry and anthropometry. We used variance decomposition procedures to partition variation of body composition into genetic and environmental components common to both sexes and to men and women separately and calculated the correlation between genetic components in men and women. Results After accounting for age, sex and inbreeding, heritability ranged from 0.39 for fat mass index to 0.84 for height. We found sex-specific genetic effects for fat percentage (fat%), lean mass, lean mass index (LMI) and fat distribution, but not for BMI and height. Genetic correlations between sexes were significantly different from 1 for fat%, lean mass, LMI, android fat, android:gynoid fat ratio and WHR, indicating that there are sex-specific genes contributing to variation of these traits. Genetic variance was significantly higher in women for the waist, hip and thigh circumference and WHR, implying that genes account for more variance of fat distribution in women than in men. Environmental variance was significantly higher in men for the android: gynoid fat ratio. Conclusions/interpretation Sex-specific genetic effects underlie sexual dimorphism in several body composition traits. The findings are relevant for studies on the relationship of body composition with common diseases like cardiovascular disease and type 2 diabetes and for genetic association studies.