Background Previous research has linked elevated low-density lipoprotein cholesterol (LDL-C) and remnant cholesterol (RC) with diabetes mellitus (DM). The present study aims to estimate the RC-related DM risk beyond LDL-C, and to investigate the extent to which the association of RC and DM is mediated via insulin resistance and inflammation. Methods We enrolled 7308 individuals without previous history of DM into the present study from the China Health and Nutrition Survey. Fasting RC was calculated as total cholesterol minus LDL-C and high-density lipoprotein cholesterol. Subjects were divided into four groups according to their LDL-C (100 mg/dL) and RC (24 mg/dL) levels to evaluate the role of LDL-C vs. RC on DM. A logistic regression analysis was then employed to evaluate the relationships between the discordant/concordant LDL-C and RC and DM. A mediation analysis was undertaken to identify potential mediators. Results Of all the participants, a total of 625 (8.55%) patients were newly diagnosed with DM. Compared to the high LDL-C/low RC group, the low LDL-C/high RC group was more common in DM patients. After a multivariate adjustment, elevated LDL-C and RC were associated with DM. Moreover, the low LDL-C/high RC group and the high LDL-C/low RC group manifested a 4.04-fold (95% CI 2.93–5.56) and 1.61-fold (95% CI 1.21–2.15) higher risk of DM, relative to those with low LDL-C/low RC. The subgroup analysis indicated that low LDL-C/high RC was more likely to be related to DM in females. Similar results were also shown when the sensitivity analyses were performed with different clinical cut-points of LDL-C. Insulin resistance and inflammation partially mediated the association between RC and DM. Conclusions Our findings provided evidence for RC beyond the LDL-C associations with DM that may be mediated via insulin resistance and the pro-inflammatory state. In addition, women are more susceptible to RC exposure-related DM.