This paper presents an investigation on the feedback controller design for active noise control headphones under the condition that the frequency responses of the primary and secondary paths corresponding to the feedback microphone do not match to the ones corresponding to the human ear. The influence of such mismatches on the performance are analyzed first, and then an optimization method is proposed to enhance the comprehensive performance at the human ear. In the proposed method, the feedback loop is constructed directly with the feedback microphone and any extra filters of the virtual sensing techniques are avoided. Cascade biquad filters are used as the controller, which is in accordance with current applications. A differential evolution algorithm was used to solve the proposed optimization problem, and the optimal parameters of the controller were found. It has been shown by the experimental results that, at the dummy head ear position, good noise reduction performance could be obtained at the low frequency band with limited noise enhancement for high frequencies, even if large frequency response mismatches exist.