The size and organization of primary visual cortex (V1) varies across individuals. Across individuals, V1 size varies more than twofold. Within individuals, cortical magnification varies with eccentricity and polar angle. Contrast sensitivity has been hypothesized to be limited by cortical resources, and specifically by the number of activated V1 neurons. Here, we quantify the relation between contrast sensitivity and V1 cortical magnification as a function of both individual observers and polar angle. We measured contrast sensitivity at four polar angle meridians in 29 observers. We then used fMRI to measure the size of V1 in the same observers, and the amount of surface area representing the same four meridians as the contrast sensitivity measurements (wedge-ROIs within 15 deg of the meridians, extending from 1 to 8 deg eccentricity). We found that: First, average contrast sensitivity per observer (averaged across polar angles) was predicted by the overall size of V1. Second, contrast sensitivity at each polar angle location was correlated with the surface area of the wedge-ROIs centered at the corresponding meridian. Third, the increases for contrast sensitivity and cortical magnification at the horizontal compared to vertical meridian (horizontal-vertical anisotropy, 'HVA') were strongly correlated: a larger HVA in contrast sensitivity corresponded to a larger HVA in cortical magnification. These results reveal that contrast sensitivity and cortical magnification co-vary across observers and demonstrate a link between perceptual polar angle asymmetries and cortical anatomy. Broadly, the results show a link between visual perception and the idiosyncratic cortical organization of V1 of neurotypical observers.