Reproductive traits are an integral part of the goals of the breeding programs that contribute to the economic success of production. Reproductive phenotypes such as litter size, number of piglets born alive, or litter weight at birth are mainly attributed to females. Thus, the maternal components can be found by default in quantitative genetics’ animal models. Still, paternal contribution to variance components should not be discarded. In this review, we indicate the importance of paternal effects in pig breeding by describing both the biology and genetics of boars’ traits, the use of (non-)genetic service sire effects in quantitative genetic models for traits measured on females, and genes involved in male reproduction. We start by describing the important biological traits of boars that have the most important effect on their reproductive abilities, i.e., sexual maturity, sperm quality, and testes parameters. Then we move to the possible environmental effects that could affect those traits of boars (e.g., feed, temperature). The main part of the review in detail describes the genetics of boars’ reproductive traits (i.e., heritability) and their direct effect on reproductive traits of females (i.e., genetic correlations). We then move to the use of both genetic and non-genetic service sire effects in quantitative models estimated as their percentage in the total variance of traits, which vary depending on the breed from 1 to 4.5% or from 1 to 2%, respectively. Finally, we focus on the description of candidate genes and confirmed mutations affecting male reproduction success: IGF2, Tgm8, ESR1, ZSWIM7, and ELMO1. In conclusion, the observed variance of paternal effects in female reproduction traits might come from various attributes of boars including biological and genetic aspects. Those attributes of boars should not be neglected as they contribute to the success of female reproductive traits.