Through this study, we aimed to determine the association of regular leisure-time physical activity (LTPA) with self-reported body mass index (BMI) and obesity risk among middle-aged and older adults in Taiwan. We conducted a cross-sectional study and reviewed the data derived from the Taiwan National Physical Activity Survey (TNPAS). Responses from 12,687 participants aged 45–108 years from the database were collected in this study. All the participants completed a standardized structured questionnaire that solicitated information regarding their demographic characteristics (age, gender, education, occupation, and self-reported health status), physical activity behaviors (regular/nonregular LTPA), and self-reported anthropometrics (height, weight, and BMI). Multiple linear and logistic regressions were used to examine the association between regular LTPA and BMI, and between regular LTPA and obesity status, respectively. Regular LTPA was associated with male gender, normal weight, excellent or good self-reported health status, and a lower rate of being underweight compared with nonregular LTPA. Regular LTPA was significant negatively associated with being underweight (OR = 0.71, p < 0.05), whereas it had no significant relationship with BMI and obesity (p > 0.05). Regular LTPA was associated with a reduced risk of being underweight among middle-aged and elderly adults in Taiwan. Further research on the relevant mechanism underlying this phenomenon is warranted.