Small non-coding RNAs (sncRNAs), being the top regulators of gene expression, have been thoroughly studied in various biological systems, including the testis. Research over the last decade has generated significant evidence in support of the crucial roles of sncRNAs in male reproduction, particularly in the maintenance of primordial germ cells, meiosis, spermiogenesis, sperm fertility, and early post-fertilization development. The most commonly studied small RNAs in spermatogenesis are microRNAs (miRNAs), PIWI-interacting RNA (piRNA), small interfering RNA (siRNA), and transfer RNA-derived small RNAs (ts-RNAs). Small non-coding RNAs are crucial in regulating the dynamic, spatial, and temporal gene expression profiles in developing germ cells. A number of small RNAs, particularly miRNAs and tsRNAs, are loaded on spermatozoa during their epididymal maturation. With regard to their roles in fertility, miRNAs have been studied most often, followed by piRNAs and tsRNAs. Dysregulation of more than 100 miRNAs has been shown to correlate with infertility. piRNA and tsRNA dysregulations in infertility have been studied in only 3–5 studies. Sperm-borne small RNAs hold great potential to act as biomarkers of sperm quality and fertility. In this article, we review the role of small RNAs in spermatogenesis, their association with infertility, and their potential as biomarkers of sperm quality and fertility.