Diabetes mellitus is a risk factor for mild cognitive impairment (MCI) and dementia. However, how the clinical characteristics of MCI patients with type 2 diabetes mellitus are linked to sarcopenia and/or its criteria remain to be elucidated. Japanese patients with type 2 diabetes mellitus were categorized into the MCI group for MoCA-J (the Japanese version of the Montreal cognitive assessment) score <26, and into the non-MCI group for MoCA-J ≥26. Sarcopenia was defined by a low skeletal mass index along with low muscle strength (handgrip strength) or low physical performance (walking speed <1.0 m/s). Univariate and multivariate-adjusted odds ratio models were used to determine the independent contributors for MoCA-J <26. Among 438 participants, 221 (50.5%) and 217 (49.5%) comprised the non-MCI and MCI groups, respectively. In the MCI group, age (61 ± 12 vs. 71 ± 10 years, p < 0.01) and duration of diabetes mellitus (14 ± 9 vs. 17 ± 9 years, p < 0.01) were higher than those in the non-MCI group. Patients in the MCI group exhibited lower hand grip strength, walking speed, and skeletal mass index, but higher prevalence of sarcopenia. Only walking speed (rather than muscle loss or muscle weakness) was found to be an independent determinant of MCI after adjusting for multiple factors, such as age, gender, body mass index (BMI), duration of diabetes mellitus, hypertension, dyslipidemia, smoking, drinking, estimated glomerular filtration rate (eGFR), HbA1c, and history of coronary heart diseases and stroke. In subgroup analysis, a group consisting of male patients aged ≥65 years, with BMI <25, showed a significant OR for walking speed. This study showed that slow walking speed is a sole determinant criterion of sarcopenia of MCI in patients with type 2 diabetes mellitus. It was suggested that walking speed is an important factor in the prediction and prevention of MCI development in patients with diabetes mellitus.