Face Recognition (FR) under varying pose, background and illumination conditions is challenging. In this paper, we propose four novel techniques, viz., Dual Pose testing (DPT), Color Based Background Removal (CBBR), Wavelet filtering and Modified Binary Particle Swarm Optimization (MBPSO), to improve the performance of a FR system. DPT is used to neutralize pose variations, CBBR removes cluttered background, Wavelet filtering is used for denoising and MBPSO, along with a modified fitness function, is used to reduce the number of selected features for recognition. Individual stages of the FR system are examined and an attempt is made to improve each stage. Experimental results show the promising performance of the proposed techniques for face recognition on four benchmark face databases, namely, Color FERET, Pointing Head Pose, CMUPIE and Extended YaleB.