Type 2 diabetes mellitus (T2DM) is well known for its adverse impacts on brain and cognition, which lead to multidimensional cognitive deficits and wildly-spread cerebral structure abnormalities. However, existing literatures are mainly focused on patients with advanced age or extended T2DM duration. Therefore, it remains unclear whether and how brain function would be affected at the initial onset stage of T2DM in relatively younger population. In current study, twelve newly-diagnosed middle-aged T2DM patients with no previous diabetic treatment history and twelve matched controls were recruited. Brain activations during a working memory task, the digit n-back paradigm (0-, 1- and 2-back), were obtained with functional magnetic resonance imaging (fMRI) and tested by repeated measures ANOVA. Whereas patients performed the n-back task comparably well as controls, significant load-by-group interactions of brain activation were found in the right dorsolateral prefrontal cortex (DLPFC), left middle/inferior frontal gyrus, and left parietal cortex, where patients exhibited hyperactivation in the 2-back but not the 0-back or 1-back condition compared to controls. Furthermore, the severity of chronic hyperglycemia, estimated by glycosylated hemoglobin (HbA1c) level, was entered into partial correlational analyses with task-related brain activations, while controlling for the real-time influence of glucose, estimated by instant plasma glucose level measured before scanning. Significant positive correlations were found between HbA1c and brain activations in the anterior cingulate cortex and bilateral DLPFC only in patients. Taken together, these findings suggest there might be a compensatory mechanism due to brain inefficiency related to chronic hyperglycemia at the initial onset stage of T2DM.