Improving reproductive performance remains a major goal in dairy cattle worldwide. Service sire has been recognized as an important factor affecting herd fertility. The main objective of this study was to reveal the genetic basis of male fertility in Italian Brown Swiss dairy cattle. Dataset included 1102 Italian Brown Swiss bulls with sire conception rate records genotyped with 454k single nucleotide polymorphisms. The analysis included whole-genome scans and gene-set analyses to identify genomic regions, individual genes and genetic mechanisms affecting Brown Swiss bull fertility. One genomic region on BTA1 showed significant additive effects. This region harbors gene RABL3 which is implicated cell proliferation and motility. Two genomic regions, located on BTA6 and BTA26, showed marked non-additive effects. These regions harbor genes, such as WDR19 and ADGRA1, that are directly involved in male fertility, including sperm motility, acrosome reaction, and embryonic development. The gene-set analysis revealed functional terms related to cell adhesion, cellular signaling, cellular transport, immune system, and embryonic development. Remarkably, a gene-set analysis also including Holstein and Jersey data, revealed significant processes that are common to the three dairy breeds, including cell migration, cell–cell interaction, GTPase activity, and the immune function. Overall, this comprehensive study contributes to a better understanding of the genetic basis of male fertility in cattle. In addition, our findings may guide the development of novel genomic strategies for improving service sire fertility in Brown Swiss cattle.