Geographic distance is a critical barrier to healthcare access, particularly for rural communities with poor transportation infrastructure who rely on non-motorized transportation. There is broad consensus on the importance of community health workers (CHWs) to reduce the effects of geographic isolation on healthcare access. Due to a lack of fine-scale spatial data and individual patient records, little is known about the precise effects of CHWs on removing geographic barriers at this level of the healthcare system. Relying on a high-quality, crowd-sourced dataset that includes all paths and buildings in the area, we explored the impact of geographic distance from CHWs on the use of CHW services for children under 5 years in the rural district of Ifanadiana, southeastern Madagascar from 2018-2021. We then used this analysis to determine key features of an optimal geographic design of the CHW system, specifically optimizing a single CHW location or installing additional CHW sites. We found that consultation rates by CHWs decreased with increasing distance patients travel to the CHW by approximately 28.1% per km. The optimization exercise revealed that the majority of CHW sites (50/80) were already in an optimal location or shared an optimal location with a primary health clinic. Relocating the remaining CHW sites based on a geographic optimum was predicted to increase consultation rates by only 7.4%. On the other hand, adding a second CHW site was predicted to increase consultation rates by 31.5%, with a larger effect in more geographically dispersed catchments. Geographic distance remains a barrier at the level of the CHW, but optimizing CHW site location based on geography alone will not result in large gains in consultation rates. Rather, alternative strategies, such as the creation of additional CHW sites or proactive care, should be considered.