Nonsyndromic orofacial clefts (OFCs) are the most common craniofacial birth defect in humans and, like many complex traits, OFCs are phenotypically and etiologically heterogenous. The phenotypic heterogeneity of OFCs extends beyond the structures affected by the cleft (e.g., cleft lip (CL) and cleft lip and palate (CLP) to other features, such as the severity of the cleft. Here, we focus on bilateral and unilateral clefts as one dimension of OFC severity. Unilateral clefts are more frequent than bilateral clefts for both CL and CLP, but the genetic architecture of these subtypes is not well understood, and it is not known if genetic variants predispose for the formation of one subtype over another. Therefore, we tested for subtype-specific genetic associations in 44 bilateral CL (BCL) cases, 434 unilateral CL (UCL) cases, 530 bilateral CLP cases (BCLP), 1123 unilateral CLP (UCLP) cases, and unrelated controls (N = 1626), using the mixed-model approach implemented in GENESIS. While no novel loci were found in subtype-specific analyses comparing cases to controls, the genetic architecture of UCL was distinct compared to BCL, with 43.8% of suggestive loci (p < 1.0×10-5) having non-overlapping confidence intervals between the two subtypes. To further understand the genetic risk factors for severity differences, we then performed a genome-wide scan for modifiers using a similar mixed-model approach and found one genome-wide significant modifier locus on 20p11 (p = 7.53×10-9), 300kb downstream of PAX1, associated with higher odds of BCL compared to UCL, which also replicated in an independent cohort (p = 0.0018) and showed no effect in BCLP (p>0.05). We further found that SNPs at this locus were associated with normal human nasal shape. Taken together, these results suggest bilateral and unilateral clefts may have differences in their genetic architecture, especially between CL and CLP. Moreover, our results suggest BCL, the rarest form of OFC, may be genetically distinct from the other OFC subtypes. This expands our understanding of genetic modifiers for subtypes of OFCs and further elucidates the genetic mechanisms behind the phenotypic heterogeneity in OFCs.