The relationship between risk of esophageal squamous cell carcinoma (ESCC) and adult height, changes in individual body mass index (BMI) and body shape is not established. We performed a large population‐based case‐control study, which enrolled a total of 1414 ESCC cases and 1989 controls in a high‐incidence area in China. Using face‐to‐face interview with a structured questionnaire, information on participants' heights, weights, and perceived body shapes at 20 years of age was collected. Additionally, data on weight and perceived body shape among the same participants 10 years prior to ascertainment were collected using the same method. Odd ratios (ORs) of ESCC risk in relation to BMI and body shape were estimated using unconditional logistic regression models. The adjusted results indicated that ESCC risk in adults rapidly rose as height increased, plateauing at 170 cm among men and 157 cm among women. Among participants who were underweight, normal weight, or thinner than body shape 4, body weight loss was associated with increased risk of ESCC, and body weight gain was associated with decreased incidence of ESCC (ORs ranging from 0.40 to 0.76). Notably, however, changes in body weight did not significantly affect ESCC risk among participants who were overweight, obese, or larger than body shape 3. Maintaining a fit body shape and a reasonable BMI is advisable and of vital importance to reduce the risk of ESCC, especially in high‐risk areas.