Breeding with a diverse population may improve potato (Solanum tuberosum subsp. tuberosum) breeding efficiency for cold chipping cultivars. The objectives of this research were to determine if breeding populations differing in assumed diversity influence the (i) frequency of cold chipping genotypes, (ii) ability to make gain in other traits subsequent to cold chipping selection, and (iii) frequency of genotypes combining high performance across many traits. Three populations or mating groups were constructed with increasing assumed diversity: Tw, Tuberosum within (intrabreeding program crosses); Tb, Tuberosum between (interbreeding program crosses); and Td, Tuberosum diverse (incorporates wild species). Potato chip color was evaluated in Year 1 on up to 20 genotypes per 4x × 4x cross. Genotypes having acceptable chip color were tested in Year 2 for tuber yield (TY), specific gravity (SG), and general tuber appearance (GTA). More Tw genotypes had acceptable chip color (1.7%) than Td (0.6%) or Tb (0.4%) and resulted from a better chip color mean and increased variation. More Td genotypes had acceptable TY (83.3%) in Year 2 than Tw (52.00%) or Tb (42.86%), and may result from increased heterozygosity. Lower numbers of effective breeding individuals in Td reduced genetic variation for all traits and reduced retention for SG and GTA. Retention after 2 yr of selection was significantly higher in Tw (0.7%) due to superior cold chipping, than Tb (0.2%) or Td (0.3%). Diversity can impact breeding efficiency, but mating groups did not predict diversity. Intermating large numbers of unrelated but adapted parents that incorporate favorable cold chipping alleles from wild species may improve breeding efficiency.