Nest construction is an energetically costly behavior displayed by males in many taxa. In some species, males construct nests and co‐breed with other males and they may construct multiple nests in a breeding season. However, little is understood about how allocation of effort within and among nests affects male reproductive success. We characterized reproductive effort of male bluehead chub (Nocomis leptocephalus), a nest‐building stream fish, in an entire breeding season in a small stream in South Carolina, USA. By using automated monitoring methods on 18 nests, we recorded time spent by 34 males during the spawning season in 2017. We then linked effort within and among nests to reproductive success, measured by the number of offspring assigned genetically to each male. A Bayesian hierarchical analysis showed that larger males spent more time constructing and maintaining a given nest, and consequently were more reproductively successful than smaller males on the same nest. Combined with aggressive behavior displayed by larger males toward smaller males, this finding suggested that reproductive effort, including agonistic interactions within nests was a determinant of reproductive success. In addition, larger nests constructed by more males led to higher male reproductive success, suggesting the importance of cooperative behaviors. Number of nests that males constructed, a measure of effort across nests, was not a predictor of reproductive success, further supporting that reproductive success varied among nests due to nest size. Our study showed that male reproductive success was determined by both aggressive and cooperative behaviors in a co‐breeding species.