Skin problems not only injure physical health but also induce psychological problems, especially for patients whose faces have been damaged or even disfigured. Using smart devices, most of the people are able to obtain convenient clinical images of their face skin condition. On the other hand, the convolutional neural networks (CNNs) have achieved near or even better performance than human beings in the imaging field. Therefore, this paper studied different CNN algorithms for face skin disease classification based on the clinical images. First, from Xiangya-Derm, which is, to the best of our knowledge, China's largest clinical image dataset of skin diseases, we established a dataset that contains 2656 face images belonging to six common skin diseases [seborrheic keratosis (SK), actinic keratosis (AK), rosacea (ROS), lupus erythematosus (LE), basal cell carcinoma (BCC), and squamous cell carcinoma (SCC)]. We performed studies using five mainstream network algorithms to classify these diseases in the dataset and compared the results. Then, we performed studies using an independent dataset of the same disease types, but from other body parts, to perform transfer learning on our models. Comparing the performances, the models that used transfer learning achieved a higher average precision and recall for almost all structures. In the test dataset, which included 388 facial images, the best model achieved 92.9%, 89.2%, and 84.3% recalls for the LE, BCC, and SK, respectively, and the mean recall and precision reached 77.0% and 70.8%. INDEX TERMS Deep learning, CNN, facial skin disease, medical image processing.