Downward continuation (DC) of the gravity potential field is an important approach used to understand and interpret the density structure and boundary of anomalous bodies. It is widely used to delineate and highlight local and shallow anomalous sources. However, it is well known that direct DC transformation in the frequency domain is unstable and easily affected by high-frequency noise. Recent deep learning applications have led to the development of image recognition and resolution enhancement using the convolutional neural network technique. A similar deep learning architecture is also suitable for training a model for the DC problem. In this study, to solve the problems in existing DC methods, we constructed a dedicated model called DC-Net for the DC problem. We fully trained the DC-Net model on 38,400 pairs of gravity anomaly data at different altitudes using a convolutional neural network. We conducted several experiments and implemented a real-world example. The results demonstrate the following. First, several validation data subset and test data prediction results indicate that the DC-Net model was sufficiently trained. Moreover, it performed better than the traditional strategy in refining the upscaling of low-resolution images. Second, we performed tests on test datasets with changing levels of noise and demonstrated that the DC-Net model is noise-resistant and robust. Finally, we used the proposed model in a real-world example, which demonstrates that the DC-Net model is suitable for solving the DC problem and delineating the detailed gravity anomaly feature near the field source. For real data processing, noise in the gravity anomaly should be reduced in advance. Additionally, we recommend noise quantification of the gravity anomaly before network training.