Exposure to lead and mercury can cause deficits in neuromotor function. Selenium and manganese are essential elements, hence both deficiency and excess could result in decreased neuromotor function. We aimed to examine hand grip strength, a marker of neuromotor function, and blood concentrations of lead, mercury, selenium, and manganese in the general U.S. population. We used data from the National Health and Nutrition Examination Survey (NHANES, 2011–2014) on 6199 participants ages 20–79 years. We assessed associations of blood concentration for these elements and grip strength with generalized regression models, and cubic splines to detect possible nonlinear relations, adjusting for confounders. The results showed that mercury and manganese were not associated with grip strength. Lead was associated with weaker grip strength in women (for 10-fold increase in lead, −2.4 kg; 95% CI: −4.2, −0.5), but not in men. Higher selenium was associated with stronger grip strength in women (8.5 kg; 95% CI: 1.9, 15.1) and men (4.6; 95% CI: −11.9, 21.0), although the association was not significant in the latter. In conclusion, lead exposure was associated with weaker grip strength in women, even at the low exposure levels in the population. Furthermore, low blood selenium level was associated with weaker grip strength, suggesting that some individuals might have selenium deficiency manifesting with poorer neuromotor function.