Automatic gender recognition is one of the frequently solved tasks in computer vision. It is useful for analysing human behaviour, intelligent monitoring or security. In this article, gender is recognized based on multispectral images of the hand. Hand (palm and back) images are obtained in the visible spectrum and thermal spectrum; then a fusion of images is performed. Some studies say that it is possible to distinguish male and female hands by some geometric features of the hand. The aim of this article is to determine whether it is possible to recognize gender by the thermal characteristics of the hand and, at the same time, to find the best architecture for this recognition. The article compares several algorithms that can be used to solve this issue. The convolutional neural network (CNN) AlexNet is used for feature extraction. The support vector machine, linear discriminant, naive Bayes classifier and neural networks were used for subsequent classification. Only CNNs were used for both extraction and subsequent classification. All of these methods lead to high accuracy of gender recognition. However, the most accurate are the convolutional neural networks VGG-16 and VGG-19. The accuracy of gender recognition (test data) is 94.9% for the palm and 89.9% for the back. Experiments in comparative studies have had promising results and shown that multispectral hand images (thermal and visible) can be useful in gender recognition.