Adult-onset progressive hearing loss is a common, complex disease with a strong genetic component. Although to date over 150 genes have been identified as contributing to human hearing loss, many more remain to be discovered, as does most of the underlying genetic diversity. Many different variants have been found to underlie adult-onset hearing loss, but they tend to be rare variants with a high impact upon the gene product. It is likely that combinations of more common, lower impact variants also play a role in the prevalence of the disease. Here we present our exome study of hearing loss in a cohort of 532 older adult volunteers with extensive phenotypic data, including 99 older adults with normal hearing, an important control set. Firstly, we carried out an outlier analysis to identify genes with a high variant load in older adults with hearing loss compared to those with normal hearing. Secondly, we used audiometric threshold data to identify individual variants which appear to contribute to different threshold values. We followed up these analyses in a second cohort. Using these approaches, we identified genes and variants linked to better hearing as well as those linked to worse hearing. These analyses identified some known deafness genes, demonstrating proof of principle of our approach. However, most of the candidate genes are novel associations with hearing loss. While the results support the suggestion that genes responsible for severe deafness may also be involved in milder hearing loss, they also suggest that there are many more genes involved in hearing which remain to be identified. Our candidate gene lists may provide useful starting points for improved diagnosis and drug development.