The immune privilege of the testes is necessary to prevent immune attacks to gamete-specific antigens and paternal major histocompatibility complex (MHC) antigens, allowing for normal spermatogenesis. However, infection and inflammation of the male genital tract can break the immune tolerance and represent a significant cause of male infertility. Different T cell subsets have been identified in mammalian testes, which may be involved in the maintenance of immune tolerance and pathogenic immune responses in testicular infection and inflammation. We reviewed the evidence in the published literature on different T subtypes (regulatory T cells, helper T cells, cytotoxic T cells, γδ T cells, and natural killer T cells) in human and animal testes that support their regulatory roles in infertility and the orchitis pathology. While many in vitro studies have indicated the regulation potential of functional T cell subsets and their possible interaction with Sertoli cells, Leydig cells, and spermatogenesis, both under physiological and pathological processes, there have been no in situ studies to date. Nevertheless, the normal distribution and function of T cell subsets are essential for the immune privilege of the testes and intact spermatogenesis, and T cell-mediated immune response drives testicular inflammation. The distinct function of different T cell subsets in testicular homeostasis and the orchitis pathology suggests a considerable potential of targeting specific T cell subsets for therapies targeting chronic orchitis and immune infertility.