ADORA2A has been shown to be responsible for the wakefulness-promoting effect of caffeine and the 1976T>C genotype (SNP rs5751876, formerly 1083T>C) to contribute to individual sensitivity to caffeine effects on sleep. We investigate the association between six single nucleotide polymorphisms (SNP) from ADORA2A and self-reported sleep characteristics and caffeine consumption in 1023 active workers of European ancestry aged 18–60 years. Three groups of caffeine consumers were delineated: low (0–50 mg/day, less than one expresso per day), moderate (51–300 mg/day), and high (>300 mg/day). We found that at caffeine levels higher than 300 mg/day, total sleep time (TST) decreased (F = 13.9, p < 0.01), with an increase of insomnia (ORa [95%CI] = 1.5 [1.1–1.9]) and sleep complaints (ORa [95%CI] = 1.9 [1.1–3.3]), whatever the ADORA2A polymorphism. Odds ratios were adjusted (ORa) for sex, age, and tobacco. However, in low caffeine consumers, lower TST was observed in the T allele compared to homozygote rs5751876 and rs3761422 C carriers. Conversely, higher TST was observed in rs2298383 T allele compared to C and in rs4822492G allele compared to the homozygote C (p < 0.05). These 4 SNPs are in strong linkage disequilibrium. Haplotype analysis confirmed the influence of multiple ADORA2a SNPs on TST. In addition, the rs2298383 T and rs4822492 G alleles were associated with higher risk of sleep complaints (Ora = 1.9 [1.2–3.1] and Ora = 1.5 [1.1–2.1]) and insomnia (Ora = 1.5 [1.3–2.5] and Ora = 1.9 [1.3–3.2). The rs5751876 T allele was associated with a decreased risk of sleep complaints (Ora = 0.7 [0.3–0.9]) and insomnia (Ora = 0.5 [0.3–0.9]). Our results identified ADORA2A polymorphism influences in the less-than-300-mg-per-day caffeine consumers. This opens perspectives on the diagnosis and pharmacology of sleep complaints and caffeine chronic consumption.