Genomic risk prediction is on the emerging path towards personalized medicine. However, the accuracy of polygenic prediction varies strongly in different individuals. In this study, based on up to 352,277 White British participants in the UK Biobank, we constructed polygenic risk scores for 15 physiological and biochemical quantitative traits after performing genome-wide association studies (GWASs). We identified 185 polygenic prediction variability quantitative trait loci (pvQTLs) for 11 traits by Levene’s test among 254,376 unrelated individuals. We validated the effects of pvQTLs using an independent test set of 58,927 individuals. A score aggregating 51 pvQTL SNPs for triglycerides had the strongest Spearman correlation of 0.185 (p-value < 1.0x10−300) with the squared prediction errors. We found a strong enrichment of complex genetic effects conferred by pvQTLs compared to risk loci identified in GWASs, including 89 pvQTLs exhibiting dominance effects. Incorporation of dominance effects into polygenic risk scores significantly improved polygenic prediction for triglycerides, low-density lipoprotein cholesterol, vitamin D, and platelet. After including 87 dominance effects for triglycerides, the adjusted R2 for the polygenic risk score had an 8.1% increase on the test set. In addition, 108 pvQTLs had significant interaction effects with measured environmental or lifestyle exposures. In conclusion, we have discovered and validated genetic determinants of polygenic prediction variability for 11 quantitative biomarkers, and partially profiled the underlying complex genetic effects. These findings may assist interpretation of genomic risk prediction in various contexts, and encourage novel approaches for constructing polygenic risk scores with complex genetic effects.