Understanding the contribution of neutral and adaptive evolutionary processes to population differences is often necessary for better informed management and conservation of rare species. In this study, we focused on Pinus torreyana Parry (Torrey pine), one of the world's rarest pines, endemic to one island and one mainland population in California. Small population size, low genetic diversity, and susceptibility to abiotic and biotic stresses suggest Torrey pine may benefit from inter-population genetic rescue to preserve the species' evolutionary potential. We leveraged reduced representation sequencing to tease apart the respective contributions of stochastic and deterministic evolutionary processes to population differentiation. We applied these data to model spatial and temporal demographic changes in effective population sizes and genetic connectivity, to assess loci possibly under selection, and evaluate genetic rescue as a potential conservation strategy. Overall, we observed exceedingly low standing variation reflecting consistently low effective population sizes across time and limited genetic differentiation suggesting maintenance of gene flow following divergence. However, genome scans identified more than 2000 SNPs candidates for divergent selection. Combined with previous observations indicating population phenotypic differentiation, this indicates that natural selection has likely contributed to population genetic differences. Thus, while reduced genetic diversity, small effective population size, and genetic connectivity between populations suggest genetic rescue could mitigate the adverse effect of rarity, divergent selection between populations indicates that genetic mixing could disrupt adaptation. Further work evaluating the fitness consequences of inter-population admixture is necessary to empirically evaluate the trade-offs associated with genetic rescue in Torrey pine.