Numerous studies have implicated the role of gross genomic rearrangements in male infertility, e.g., constitutional aneuploidy, translocations, inversions, Y chromosome deletions, elevated sperm disomy, and DNA damage. The primary purpose of this paper is to review male fertility studies associated with such abnormalities. In addition, we speculate whether altered nuclear organization, another chromosomal/whole genome-associated phenomenon, is also concomitant with male factor infertility. Nuclear organization has been studied in a range of systems and implicated in several diseases. For many applications the measurement of the relative position of chromosome territories is sufficient to determine patterns of nuclear organization. Initial evidence has suggested that, unlike in the more usual ‘size-related’ or ‘gene density-related’ models, mammalian (including human) sperm heads display a highly organized pattern including a chromocenter with the centromeres located to the center of the nucleus and the telomeres near the periphery. More recent evidence, however, suggests there may be size- and gene density-related components to nuclear organization in sperm. It seems reasonable to hypothesize therefore that alterations in this pattern may be associated with male factor infertility. A small handful of studies have addressed this issue; however, to date it remains an exciting avenue for future research with possible implications for diagnosis and therapy.