There is evidence for contribution of inherited factors to prostate cancer, and more specifically to lethal prostate cancer, but few responsible genes/variants have been identified. We examined genetic sequence data for 51 affected cousin pairs who each died from prostate cancer and who were members of high-risk prostate cancer pedigrees in order to identify rare variants shared by the cousins as candidate predisposition variants. Candidate variants were tested for association with prostate cancer risk in UK Biobank data. Candidate variants were also assayed in 1195 additional sampled Utah prostate cancer cases. We used 3D protein structure prediction methods to analyze structural changes and provide insights into mechanisms of pathogenicity. Almost 4000 rare (<0.005) variants were identified as shared in the 51 affected cousin pairs. One candidate variant was also significantly associated with prostate cancer risk among the 840 variants with data in UK Biobank, in the gene LRBA (p = 3.2 × 10−5; OR = 2.09). The rare risk variant in LRBA was observed to segregate in five pedigrees. The overall predicted structures of the mutant protein do not show any significant overall changes upon mutation, but the mutated structure loses a helical structure for the two residues after the mutation. This unique analysis of closely related individuals with lethal prostate cancer, who were members of high-risk prostate cancer pedigrees, has identified a strong set of candidate predisposition variants which should be pursued in independent studies. Validation data for a subset of the candidates identified are presented, with strong evidence for a rare variant in LRBA.