A nonparametric method to define a pixel neighborhood in catadioptric images is presented. The method is based on an accurate modeling of the mirror shape by mean of polarization imaging. Unlike most processing methods existing in the literature, this method is nonparametric and enables us to respect the catadioptric image's anamorphosis. The neighborhood is directly derived from the two polarization parameters: the angle and the degree of polarization. Regardless of the shape of the catadioptric sensor's mirror (including noncentral configurations), image processing techniques such as image derivation, edge detection, interest point detection, as well as image matching, can be efficiently performed.