Ear detection in facial images under varying pose, background and occlusion is a challenging issue. This paper proposes an entropy-cum-Hough-transform-based approach for enhancing the performance of an ear detection system, employing the unique combination of hybrid ear localizer (HEL) and ellipsoid ear classifier (EEC). By exploiting the entropic properties of the ear, as well as its ellipsoid structure, the HEL identifies the most probable location of the ear. To curb false ear acceptances by the HEL, the EEC verifies the actual presence of an ear in facial images, by attempting to fit an ellipse on the localized ear. EEC performs this by using the ellipsoid parametric set optimized by the evolutionary ellipsoid particle swarm optimization technique. The amount of success met by the EEC in doing so, i.e., the 'goodness of fit', is used to verify the presence of an ear. Experiments have been carried out on five benchmark face databases, namely Color FERET, Pointing Head Pose, UMIST, CMU-PIE, and FEI. The results show the efficiency and robustness of both HEL and EEC working individually and as a system, for enhanced ear detection.