Background: Diabetes and physical inactivity are prevalent worldwide. Risk of diabetes is known to be related with insufficient physical activity (PA), but associations with the respective dimensions of PA is unclear. Objective: To describe the patterns of physical activity among Chinese middle- and older-aged individuals and figure out their associations with diabetes risk in different dimensions. Methods: Extracting self-reported data from China Health and Retirement Longitudinal Study (CHARLS, 2015), this study included 6196 participants. Multivariate logistic regression was conducted to determine the association between diabetes risk and PA dimensions such as intensity, frequency, duration, and volume. Results: Concerning frequency, lower diabetes risk was associated with performing vigorous PA at any frequency overall. For duration, smaller odds of diabetes were observed in performing vigorous PA 2–4 h/day (OR 0.46, 95%CI 0.30 to 0.71), moderate PA ≥4 h/day (OR 0.59, 95%CI 0.42 to 0.82) and light PA ≥4 h/day (OR 0.59, 95%CI 0.41 to 0.85) overall. For volume, lower diabetes risk was associated with performing moderate-to-vigorous PA (MVPA) ≥2250 METs/week (OR 0.58, 95%CI 0.42 to 0.81) in middle-aged group (45–64 years), whereas no significant associations between MVPA and diabetes risk were found in older aged group (≥65 years). Conclusions: Our results revealed that physical inactivity is prevalent in China, with a greater proportion in the diabetes group. Lower risk of diabetes was associated with higher frequency, longer duration and longer volume of PA at higher intensity in middle-aged respondents and similar associations at lower intensity for the older adults. Additionally, further well-designed prospective studies are needed to confirm our findings.