With the recent development of the technology, it is seen that there is a significant increase in the studies on the analysis of human faces. Through the automatic analysing of the faces, it is possible to know the gender, emotional state, and even the identity of the people from an image. Of them, identity or face recognition has became the most important task which has been studied for a long time now as it is crucial to take measurement for public security, credit card verification, criminal identification, and the like. In this study, we have proposed an evolutionary-based framework that relies on genetic programming algorithm to evolve a binary-and multi-label image classifier program for gender classification, facial expression recognition, and face recognition tasks. The performance of the evolved programs has been compared with convolutional neural network, one of the most popular deep learning algorithms. The comparative results show that the proposed framework could better performance than the competitor algorithm. Therefore, it has been introduced to the research community as a new binary-and multi-label image classifier framework.