Physiological signals are strongly related to a person’s state of health and carry information about the human body. For example, by ECG, it is possible to obtain information about cardiac disease, emotions, personal identification, and the sex of a person, among others. This paper proposes the study of the heartbeat from a soft-biometric perspective to be applied to smartphone unlocking services. We employ the user heartbeat to classify the individual by sex (male, female) with the use of Deep Learning, reaching an accuracy of 94.4% ± 2.0%. This result was obtained with the RGB representation of the union of the time-frequency transformation from the pseudo-orthogonal X, Y, and Z bipolar signals. Evaluating each bipolar contribution, we found that the XYZ combination provides the best category distinction using GoogLeNet. The 24-h Holter database of the study contains 202 subjects with a female size of 49.5%. We propose an architecture for managing this signal that allows the use of a few samples to train the network. Due to the hidden nature of ECG, it does not present vulnerabilities like public trait exposition, light/noise sensibility, or learnability compared to fingerprint, facial, voice, or password verification methods. ECG may complement those gaps en route to a cooperative authentication ecosystem.