Introduction: Type 2 diabetes mellitus is one of the major causes of increasing morbidity worldwide. Effective screening is carried out routinely for diabetic retinopathy, neuropathy, and nephropathy. Of late, studies have reported that cognitive decline can occur in people with diabetes, which could go undetected for a long period, and hence routine screening could be warranted. Methodology: Our objective was to study the prevalence of previously unknown mild cognitive impairment (MCI) in type 2 diabetic patients visiting a tertiary care center with the Montreal Cognitive Assessment (MoCA) test and to study the correlations of HbA1c, fasting blood sugar (FBS), postprandial blood sugar (PPBS), age, and duration of diabetes with the MoCA scores. Seventy patients with type 2 diabetes mellitus were included in the study. Patients with MoCA scores ≥26 were considered to have normal cognition (NC) and those with <26 MCI. Results: MCI was noted in 38 (54.29%) type 2 diabetes mellitus patients and NC in 32 (45.71%). Those with MCI had higher HbA1c (8.79 ± 1.85 vs. 7.78 ± 1.60), higher FBS (177.05 ± 62.48 vs. 149.38 ± 54.38), and PPBS (282.03 ± 85.61 vs. 214.50 ± 82.43), which were statistically significant. The cognitive domains of executive function, naming, attention, language, and memory showed a statistically significant difference between those with MCI and NC. There were no differences in the mean age, duration of diabetes, and educational status between the groups. Conclusion: The high prevalence of MCI in type 2 diabetic patients highlights the importance of implementing routine cognitive testing. The correlation of cognitive impairment with poor glucose control needs further studies to find out whether improving glycemic control will help improve cognition.