Identifying genetic risk factors for highly heterogeneous disorders like epilepsy remains challenging. Here, we present the largest whole-exome sequencing study of epilepsy to date to investigate rare variants that confer risk for a spectrum of epilepsy syndromes. With an unprecedented sample size of >54,000 human exomes, composed of 20,979 deep-phenotyped patients with epilepsy and 33,444 controls, we replicate previous gene findings at exome-wide significance; using a hypothesis-free approach, we identify potential novel associations. Most discoveries are specific to a particular subtype of epilepsy, highlighting distinct genetic contributions to different epilepsies. Combining evidence from rare single nucleotide/short indel-, copy number-, and common variants, we find convergence of different genetic risk factors at the level of individual genes. Further comparing to other exome-sequencing studies, we implicate shared rare variant risk between epilepsy and other neurodevelopmental disorders. Our study also demonstrates the value of collaborative sequencing and deep-phenotyping efforts, which will continue to unravel the complex genetic architecture underlying the heterogeneity of epilepsy.