SummaryCell polarity is essential for many cellular functions including division and cell-fate determination. Although RhoGTPase signaling and vesicle trafficking are both required for the establishment of cell polarity, the mechanisms by which they are coordinated are unclear. Here, we demonstrate that the yeast RhoGAP (GTPase activating protein), Bem3, is targeted to sites of polarized growth by the endocytic and recycling pathways. Specifically, deletion of SLA2 or RCY1 led to mislocalization of Bem3 to depolarized puncta and accumulation in intracellular compartments, respectively. Bem3 partitioned between the plasma membrane and an intracellular membrane-bound compartment. These Bem3-positive structures were polarized towards sites of bud emergence and were mostly observed during the pre-mitotic phase of apical growth. Cell biological and biochemical approaches demonstrated that this intracellular Bem3 compartment contained markers for both the endocytic and secretory pathways, which were reminiscent of the Spitzenkörper present in the hyphal tips of growing fungi. Importantly, Bem3 was not a passive cargo, but recruited the secretory Rab protein, Sec4, to the Bem3-containing compartments. Moreover, Bem3 deletion resulted in less efficient localization of Sec4 to bud tips during early stages of bud emergence. Surprisingly, these effects of Bem3 on Sec4 were independent of its GAP activity, but depended on its ability to efficiently bind endomembranes. This work unveils unsuspected and important details of the relationship between vesicle traffic and elements of the cell polarity machinery: (1) Bem3, a cell polarity and peripherally associated membrane protein, relies on vesicle trafficking to maintain its proper localization; and (2) in turn, Bem3 influences secretory vesicle trafficking.