In many bird species, physical aggression between males become more frequent during the female’s fertile period, as female encounters with extra-pair males are more frequent and can entail paternity losses. Male aggressiveness during this stage has been proposed as crucial for ensuring male reproductive success. Thus, plumage ornaments could represent honest signals of individual quality that could reflect the aggressiveness of paired territorial males. Furthermore, male aggressiveness could be related to mate quality or defensive capacity. We quantified extra-pair paternity in the broods and investigated the association of male and female traits with the aggressive behaviour of territorial paired males in a Spanish population of Pied Flycatchers (Ficedula hypoleuca), where territorial intrusions were simulated during the female fertile period by placing a taxidermic male mount close to the nest. We predicted that (1) more aggressive males should better protect their mates from intruding males and thereby reduce their paternity losses, (2) males with larger white patches and higher UV reflectance of wing patches should respond more strongly to intrusions, and (3) that males should be more aggressive when mated with higher quality females. We found evidence that males that responded less intensely to a territorial intrusion suffered a higher paternity loss, which offers strong support to the basic tenet of the theory of territoriality as paternity defence. Moreover, both the level of male aggressiveness and control of the territory increased with male UV reflectance of wing patches. Finally, we found, contrary to our prediction, that males were less aggressive when mated with more ornamented females.