Human handwriting is used to investigate human characteristics in various applications, including but not limited to biometric authentication, personality profiling, historical document analysis, and forensic investigations. Gender is one of the most distinguishing characteristics of human beings. From this point forth, we propose a novel end-to-end model based on Convolutional Neural Network (CNN) that automatically extracts features from a given handwritten sample, which contains both handwritten text and numerals unlike the related work that uses only handwritten text, and classifies its owner’s gender. In addition to proposing a novel model, we introduce a new dataset that consists of 530 gender-labeled Turkish handwritten samples since, to the best of our knowledge, there does not exist a public gender-labeled Turkish handwriting dataset. Following an exhaustive process of hyperparameter optimization, the proposed CNN featured the most optimal hyperparameters and was both trained and evaluated on this dataset. According to the experimental result, the proposed novel model obtained an accuracy as high as 74.46%, which overperformed the state-of-the-art baselines and is promising on such a task that even humans could not have achieved highly-accurate results for, as of yet.