Background To elucidate the clinical characteristics of type 2 diabetes mellitus patients with mild cognitive impairment (MCI) and to examine whether diagnosis of sarcopenia and/or its criterion are explanatory factors for MCI Methods Patients with type 2 diabetes mellitus were categorized into the MCI group for MoCA-J (the Japanese version of the Montreal cognitive assessment) score <26, and into the non-MCI group for MoCA-J ≥26. Sarcopenia was defined by a low skeletal mass index along with low muscle strength (handgrip strength) or low physical performance (walking speed <1.0 m/s). Univariate and multivariate-adjusted odds ratio models were used to determine the independent contributors for MoCA-J <26. Results Among 438 participants, 221 (50.5%) and 217 (49.5%) comprised the non-MCI and MCI groups, respectively. In the MCI group, age (61 ± 12 vs. 71 ± 10 years, p < 0.01) and duration of diabetes (14 ± 9 vs. 17 ± 9 years, p < 0.01) were higher than those in the non-MCI group. Patients in the MCI group exhibited lower hand grip strength, walking speed, and skeletal mass index, but higher prevalence of sarcopenia. Only walking speed (rather than muscle loss or muscle weakness) was found to be an independent determinant of MCI after adjusting for multiple factors, such as age, gender, BMI, duration of diabetes, hypertension, dyslipidemia, smoking, drinking, eGFR, HbA1c, and history of coronary heart diseases and stroke. In subgroup analysis, a group consisting of male patients aged ≥65 years, with BMI <25, showed a significant OR for walking speed. Conclusions This is the first study to show that slow walking speed is a determinant for the presence of MCI in patients with type 2 diabetes. It was suggested that walking speed is an important factor in the prediction and prevention of MCI development in patients with diabetes mellitus.