Background Acarbose is one of the optimal drugs for patients with the first diagnosis of type 2 diabetes mellitus (T2DM). But what kind of emerging patients has the best therapeutic response to acarbose therapy has never been reported. To this end, we investigated predictors of acarbose therapeutic efficacy in newly diagnosed T2DM patients in China. Methods A total of 346 T2DM patients received acarbose monotherapy for 48 weeks as part of participating in the Study of Acarbose in Newly Diagnosed Patients with T2DM in China (MARCH study) from November 2008 to June 2011. Change in glycated hemoglobin (ΔHbA1c) served as a dependent variable while different baseline variables including sex, age, disease duration, weight, body mass index (BMI), systolic blood pressure (SBP), diastolic blood pressure (DBP), HbA1c, fasting plasma glucose (FPG), 2-h postprandial blood glucose (2 h PG), fasting insulin (FINS), 2-h postprandial insulin (2 h INS), early insulin secretion index (IGI), homeostasis model assessment of insulin resistance index (HOMA-IR), homeostasis model assessment of beta cell function (HOMA-B), area under the curve (AUC) of glucagon, insulin and GLP-1 were assessed as independent predictors. Step-wise multiple linear regression was employed for statistical analysis. Results The results suggested that independent predictors of ΔHbA1c at 12 weeks included baseline body weight (β = − 0.012, P = 0.006), DBP (β = 0.010, P = 0.047), FPG (β = 0.111, P = 0.005) and 2 h PG (β = 0.042, P = 0.043). Independent predictors of ΔHbA1c at 24 weeks included disease duration (β = 0.040, P = 0.019) and FPG (β = 0.117, P = 0.001). Finally, independent predictor of ΔHbA1c at 48 weeks was disease duration (β = 0.038, P = 0.046). Conclusions Acarbose may be more effective in newly diagnosed T2DM patients with low FPG, low 2 h PG and obesity. The earlier T2DM is diagnosed and continuously treated with acarbose, the better the response to therapy.