Worldwide, one in five men aged over 50 years will experience osteoporosis or a clinical bone fracture, with a greater fracture‐related mortality rate than women. However, the genetic etiology of osteoporosis in men is still poorly understood. We aimed to identify the genetic variants and candidate genes associated with extremely low or high BMD for a better understanding of the biology underlying low bone density that may point to potential therapeutic targets for increasing bone mass. Subjects from the Osteoporotic Fractures in Men Study (MrOS) cohort were evaluated by age and BMI‐adjusted total hip BMD. Those with BMD values 3 SDs away from the mean were selected and the remaining individuals whose adjusted BMD ranked at the highest or lowest 100 were included. Men with the lowest adjusted BMD (N = 98) and highest adjusted BMD (N = 110) were chosen for exome sequencing. Controls (N = 82) were men of Northern and Western European descent from the US Utah population of the 1000 Genomes Project. Fisher's exact test was performed to compare low‐ or high‐BMD subjects with controls for single‐gene associations. Additionally, sets of candidate genes causative of heritable disorders of connective tissue, including osteogenesis imperfecta (OI) and Ehlers‐Danlos syndrome (EDS), were grouped for multigene and mutation burden analyses. No single‐gene associations with rare variants were found for either the low BMD group (33 genes) or high BMD group (18 genes). In the group of OI genes, we detected a significant threefold increased accumulation of rare variants in low‐BMD subjects compared with controls (p = 0.009). Additionally, genes associated with EDS had a twofold increased frequency in low‐BMD subjects compared with controls (p = 0.03). These findings reveal a rare variant burden in OI and EDS disease genes at low BMD, which suggests a potential gene‐panel approach to screen for multivariant associations in larger cohorts. © 2019 The Authors. JBMR Plus published by Wiley Periodicals, Inc. on behalf of American Society for Bone and Mineral Research.