Men and women with type 2 diabetes mellitus (T2DM) have higher risk of hip fracture, but the mechanisms are not fully understood. We aimed to investigate how T2DM, glucose, and insulin were associated with femoral bone mineral density (BMD), bone mineral area (BMA), and bone turnover markers. We used two cross-sectional cohorts: the Uppsala Longitudinal Study of Adult Men (ULSAM, n = 452, mean age 82 years) and the Swedish Mammography Cohort Clinical (SMCC, n = 4713, mean age 68 years). We identified men and women with normal fasting glucose (NFG), impaired fasting plasma glucose (IFG), and T2DM. BMD and BMA at the total hip and femoral shaft were measured using dual energy X-ray absorptiometry (DXA). Bone turnover markers; CrossLaps and osteocalcin were measured in women. Linear regression models were applied. Men and women showed a progressively higher BMD following the clinical cutoffs of fasting glucose from NFG to IFG to T2DM. In contrast, there was a progressively lower BMA. Men and women with T2DM, compared to those with NFG, had lower BMA at the total hip (− 1.7%; 95% CI − 3.2, − 0.2 and − 1.0%; 95% CI − 1.6, − 0.4) and the femoral shaft (− 2.0%; 95% CI − 3.5, − 0.4 and − 0.6%; 95% CI − 1.2, − 0.01), respectively. T2DM was associated with lower concentrations of CrossLaps (− 8.1%; 95% CI − 12.7, − 3.6) and osteocalcin (− 15.2%; 95% CI − 19.0, − 11.2). These cross-sectional results indicate that those with T2DM have smaller bone area and lower bone turnover, which could increase the risk of hip fracture.Electronic supplementary materialThe online version of this article (10.1007/s00223-018-0446-9) contains supplementary material, which is available to authorized users.