Cell contacts provide spatial cues that polarize early embryos and epithelial cells. The homophilic adhesion protein E-cadherin is required for contact-induced polarity in many cells. However, it is debated whether E-cadherin functions instructively as a spatial cue, or permissively by ensuring adequate adhesion so that cells can sense other contact signals. In C. elegans, contacts polarize early embryonic cells by recruiting the RhoGAP PAC-1 to the adjacent cortex, inducing PAR protein asymmetry. Here we show that HMR-1/E-cadherin, which is dispensable for adhesion, functions together with HMP-1/α-catenin, JAC-1/p120 catenin, and the previously uncharacterized linker PICC-1/CCDC85/DIPA to bind PAC-1 and recruit it to contacts. Mislocalizing the HMR-1 intracellular domain to contact-free surfaces draws PAC-1 to these sites and depolarizes cells, demonstrating an instructive role for HMR-1 in polarization. Our findings identify an E-cadherin-mediated pathway that translates cell contacts into cortical polarity by directly recruiting a symmetry-breaking factor to the adjacent cortex.