Community health workers (CHWs) have a unique position between communities and the health sector. The strength of CHWs' relationships with both sides influences their motivation and performance. This qualitative comparative study aimed at understanding similarities and differences in how relationships between CHWs, communities and the health sector were shaped in different Sub-Saharan African settings. The study demonstrates a complex interplay of influences on trust and CHWs' relationships with their communities and actors in the health sector. Mechanisms influencing relationships were feelings of (dis)connectedness, (un)familiarity and serving the same goals, and perceptions of received support, respect, competence, honesty, fairness and recognition. Sometimes, constrained relationships between CHWs and the health sector resulted in weaker relationships between CHWs and communities. The broader context (such as the socio-economic situation) and programme context (related to, for example, task-shifting, volunteering and supervision) in which these mechanisms took place were identified. Policy-makers and programme managers should take into account the broader context and could adjust CHW programmes so that they trigger mechanisms that generate trusting relationships between CHWs, communities and other actors in the health system. This can contribute to enabling CHWs to perform well and responding to the opportunities offered by their unique intermediary position.