Although it is well recognized that the strength of plant–herbivore interactions can vary with the plant sex, the distance, and the density of conspecific neighbors, no study has yet assessed their combined influence. Here, we filled this knowledge gap by focusing on the dioecious palm Chamaeropshumilis L., and its two main herbivores, the invasive moth Paysandisiaarchon Burmeister and the feral goat Caprahircus L. We evaluated levels and spatial patterns of herbivory, as well as those of plant size and number of inflorescences in two palm populations in Mallorca (Balearic Islands, Spain). Our spatial point pattern analyses revealed that palms not affected by moth herbivory or goat florivory were spatially aggregated, goats fed more strongly upon inflorescences in palms with more neighbors, but they consumed more leaves in isolated palms. Interestingly, we could reveal for the first time that plant sex is a key plant trait modulating neighborhood effects. For instance, whereas aggregated female palms experienced lower intensity of goat florivory than isolated ones, male palms showed the opposite pattern. Palm size and number of inflorescences also showed sex-related differences, suggesting that sexual dimorphism is a key driver of the observed neighborhood effects on herbivory. Our study highlights the importance of considering relevant plant traits such as sex when investigating plant neighborhood effects, calling for further research to fully understand the dynamics governing plant–herbivore interactions in dioecious systems.