We performed a common garden experiment using parental, F1, F2, and backcross willow hybrids to test the hypothesis that hybrid willows experience breakdown of resistance to herbivores. After exposing plants to herbivores in the field, we measured the densities/damage caused by 13 insect herbivores and one herbivorous mite. Using joint‐scaling tests, we determined the contribution of additive, dominance, and epistasis to variation in susceptibility to herbivores (measured either as density or damage level) among the six genetic classes. We found the genetic architecture of susceptibility/resistance in the parental species to be complex, involving additive, dominance, and epistasis for each herbivore species. Although genic interactions altered plant susceptibility for each of the 14 herbivores, three distinct patterns of response of herbivores to hybrids were expressed. One pattern, observed in four herbivore species, supported the hypothesis of breakdown of resistance genes in recombinant hybrids. A second pattern, shown by six other herbivore species, supported the hypothesis of hybrid breakdown of host recognition genes. In other words, epistatic interactions for host recognition traits (probably oviposition/feeding stimulants or attractants) appeared to be important in determining herbivore abundance for those six species. The final patterns supported a structure of dominance, either for host recognition traits (in the case of three herbivore species) or for host resistance traits (for one herbivore species). The combination of differing responses of herbivore species, including members of the same genus and tribe, and the ubiquitous importance of epistasis suggests that many genes affect herbivore resistance in this hybrid willow system.