Recent advances in face recognition are mostly based on deep learning methods that require large datasets for training. For smaller datasets, we propose a method that combines Gabor feature extraction and aggressive kernel selection to achieve low error rates while keeping computational cost at a minimum. The paper compares the proposed method against traditional feature selection approaches in terms of the recognition accuracy and model compression and show that the proposed method can achieve the same or higher accuracy with significantly lower computational cost. Moreover, we evaluated combining multiple feature selection algorithms to derive our proposed kernel selection method achieving an error rate of 0.025 on the Yala face dataset.