Background: Recently, we have shown that extremely high high-density lipoprotein cholesterol (HDL-C), which was observed mostly in heavy drinkers, was associated with the incidence of diabetes. However, the observed association was influenced by the consumption of alcohol. Furthermore, it is unknown whether impaired glucose metabolism exists in people with extremely high HDL-C, regardless of their alcohol consumption. Therefore, we addressed this issue in people who did not have a habit of drinking alcohol. Methods: In this community-based cross-sectional study, we included 177,034 participants (40–74 years old) who reported being nondrinkers. We investigated levels of HbA1c, fasting plasma glucose (FPG), HDL-C, and clinical parameters according to 11 levels of HDL-C concentration from 20 to 120 mg/dL or over. Results: A total of 6112 participants with HDL-C ≥ 100 mg/dL (3.5%) showed a better lipid profile, higher prevalence amongst women, more habitual exercise, a lower prevalence of smoking, and lower body mass index (BMI). Compared with an HDL-C of 70–79 mg/dL, HDL-C ≤ 69 mg/dL (except an HDL-C of 20–29 mg/dL) and HDL-C ≥ 90 mg/dL were significantly associated with a high HbA1c of ≥6.0%, independently of confounding factors. This finding was distinctly demonstrated in women. Similar trends were observed when high HbA1c was replaced with high FPG (≥110 mg/dL). Conclusions: Our study demonstrated that impaired glucose metabolism may exist in people with extremely high HDL-C and who hardly drink alcohol.