Evaluating thyroid nodules to rule out malignancy is a very common clinical task. Image-based clinical and machine learning risk stratification schemas rely on the presence of thyroid nodule high-risk sonographic features. However, this approach is less suitable for diagnosing malignant thyroid nodules with a benign appearance on ultrasound. In this study, we developed thyroid cancer polygenic risk scoring (PRS) to complement deep learning analysis of ultrasound images. When the output of the deep learning model was combined with thyroid cancer PRS and genetic ancestry estimates, the area under the receiver operating characteristic curve (AUROC) of the benign vs. malignant thyroid nodule classifier increased from 0.83 to 0.89 (DeLong, p-value = 0.007). The combined deep learning and genetic classifier achieved a clinically relevant sensitivity of 0.95, 95 CI [0.88-0.99], specificity of 0.63 [0.55-0.70], and positive and negative predictive values of 0.47 [0.41-0.58] and 0.97 [0.92-0.99], respectively. An improved AUROC was consistent in ancestry-stratified analysis in Europeans (0.83 and 0.87 for deep-learning and deep learning combined with PRS classifiers, respectively). An elevated PRS was associated with a greater risk of thyroid cancer structural disease recurrence (ordinal logistic regression, p-value = 0.002). This study demonstrates that augmenting ultrasound image analysis with PRS improves diagnostic accuracy, paving the way for developing the next generation of clinical risk stratification algorithms incorporating inherited risk for developing thyroid malignancy.