Electronic health records (EHRs) are often incomplete and inaccurate, reducing the power of genome-wide association studies (GWAS). Moreover, the variables within these records are often represented in binary codes, masking variation in disease severity among individuals. For some diseases, such as knee osteoarthritis (OA), radiographic assessment is the primary means of diagnosis and can be performed directly from medical images. In this work, we trained a deep learning model (DL-binary) to ascertain knee OA cases from anteroposterior (AP) dual-energy absorptiometry (DXA) scans and achieved clinician level performance. Applying this model across 29,257 individuals from the UK Biobank (UKB), we identified 2,603 (240%) more cases than currently diagnosed in the ICD-10 record. Individuals diagnosed as cases by DL-binary had higher rates of self-reported knee pain, knee pain for longer durations and with increased severity compared to control individuals. We trained another deep learning model to measure the minimum knee joint space width (mJSW), a quantitative phenotype linked to knee OA severity. Despite the DL-binary phenotype and mJSW being highly genetically correlated (92%), the heritability of mJSW was an order of magnitude greater than the ICD-10 code M17 or DL-binary phenotypes. In a GWAS run on mJSW, we identified 18 genome-wide significant loci, as opposed to 1 and 6 at the same sample size using either case-control (DL-binary and ICD-10 code M17) phenotype. This improved power also translated to better polygenic risk score (PRS) prediction for knee OA diagnosis in a holdout dataset of 371,686 individuals. We also show that reduced mJSW, but neither case-control phenotype is associated with increased risk of adult fractures, a leading cause of injury-related death in older individuals. For diseases with radiographic diagnosis, our results demonstrate the enormous potential for using deep learning to phenotype at biobank scale, both for improving power for gene discovery and for epidemiological analysis.