The foundation of potato (Solanum tuberosum) breeding and the development of new potato varieties is the crossing of parents to produce seeds that contain new combinations of alleles that may be superior to those in either parent. In this review, we begin by highlighting a signature problem of potato breeding, the widespread lack of fertility observed in many lines. We summarize normal development of male and female reproductive structures, fertilization and embryo growth. Various manifestations of male and female sterility are then described. These include defects of meiosis and mitosis, bud abscission, competition between reproductive structures and tubers for resources, and cytoplasmic male sterility. Effects of environmental determinants, such as temperature and light intensity, on fertility are discussed. Finally, suggested best practices that promote fertility are provided as a guide for those conducting breeding and genetics work with potato.