(1) Objective: To assess the association of environmental chemical factors with osteopenia and/or bone fractures. (2) Methods: All data were extracted from the National Health and Nutrition Survey (NHANES) 2017–2018 of American adults aged 20–59 years old; invalid data were excluded based on dual-energy X-ray absorptiometry. For the ultimate valid data set, multivariate logistic regression models were applied to evaluate the association of environmental chemical factors with osteopenia and bone fractures. (3) Results: The valid dataset was obtained from 2640 individuals, who completed a questionnaire of demographic characteristics. Urinary manganese and monomethylarsonic acid were positively associated with osteopenia in American adults, but not bone fracture. However, several environmental factors (e.g., arsenous acid, arsenocholine, dimethylarsinic acid, and 2-thioxothiazolidine-4-carboxylic acid) did not affect bone mineral density, but were significantly associated with bone fracture. (4) Conclusions: Multiple environmental chemical factors significantly affect bone mass or fracture risk. However, the risk of environmental chemical factors on fractures is independent of osteopenia in US Adults. The influence of environmental chemical factors on bone quality should be considered and monitored.