The skin surface is composed of a network-like microstructure comprising wrinkles. Observing and analyzing the microstructure of the skin that changes with the skin condition and aging are simple, stable, and accurate evaluation methods for skin diagnosis. However, the skin surface includes various morphological and topological changes, depending on the individual or the degree of aging. It is difficult to accurately extract and analyze a skin microstructure including these changes. Therefore, we perform skin microstructure segmentation and aging analysis by using convolutional neural network (CNN) models. First, we propose a fusion UNet model to extract the skin microstructure. We compare and evaluate the segmentation performance by using an image processing method and deep learning models. Next, we classify skin aging based on the skin microstructure. For the classification, we use four mobile CNN models: NASNet-Mobile, MobileNetV2, MobileNetV3-Small, and EfficientNet-B0. Subsequently, we compare and evaluate their classification performances. Results show that the segmentation images of the fusion U-Net are most similar to the ground truth, and the fusion U-Net model can detect fine wrinkles that are difficult to identify by the naked eye. In the microstructure-based classification of skin aging, MobileNetV3-Small exhibits the best performance with an accuracy of 94%. The proposed method facilitates an objective and quantitative analysis of the skin surface with more diverse aging characteristics. Consequently, the association between skin aging and skin microstructure changes is confirmed. Our study can be utilized in the diagnostic studies on various skin characteristics, including skin texture, anisotropy, and roughness. The proposed method can also be applied to a mobile-based self-diagnosis system.