In nature, host-parasite/pathogen relationships are embedded in a network of ecological interactions that have the potential to shape the evolutionary trajectories of shared pathogens. Understanding this community context of infectious disease evolution is important for wildlife, agricultural, and human systems alike -- illustrated, for example, by the increasing risk of zoonotic disease emergence. We introduce an eco-evolutionary model that examines ecological feedbacks across a range of host-host interactions. Specifically, we analyze a model of the evolution of virulence of a pathogen infecting hosts who themselves exhibit competitive, mutualistic, or exploitative relationships. We find that pathogen specialism is necessary for inter-host interactions to impact parasite evolution. An important general result is that increasing competition between hosts leads to higher shared pathogen virulence, while increasing mutualism leads to lower virulence. Across a range of scenarios, the nature of pathogen specialization is critical to the outcome -- for instance, if hosts only differ in initial susceptibility to infection, there is no impact of host-host interactions on virulence evolution. In contrast, specialization in terms of onward transmission, host tolerance, or intra-host pathogen growth rate critically impact the evolution of virulence. For example, stronger specialism in transmission selects for lower virulence, while stronger specialism in tolerance and growth rate selects for higher virulence. Our work provides testable hypotheses for multi-host disease systems, predicts how changing interaction networks may impact the evolution of virulence, and broadly demonstrates the importance of looking beyond pairwise relationships to understand evolution in realistic natural contexts.