The prediction of soil properties at different depths is an important research topic for promoting the conservation of black soils and the development of precision agriculture. Mid-infrared spectroscopy (MIR, 2500–25000 nm) has shown great potential in predicting soil properties. This study aimed to explore the ability of MIR to predict soil organic matter (OM) and total nitrogen (TN) at five different depths with the calibration from the whole depth (0–100 cm) or the shallow layers (0–40 cm) and compare its performance with visible and near-infrared spectroscopy (vis-NIR, 350–2500 nm). A total of 90 soil samples containing 450 subsamples (0–10 cm, 10–20 cm, 20–40 cm, 40–70 cm, and 70–100 cm depths) and their corresponding MIR and vis-NIR spectra were collected from a field of black soil in Northeast China. Multivariate adaptive regression splines (MARS) were used to build prediction models. The results showed that prediction models based on MIR (OM: RMSEp = 1.07–3.82 g/kg, RPD = 1.10–5.80; TN: RMSEp = 0.11–0.15 g/kg, RPD = 1.70–4.39) outperformed those based on vis-NIR (OM: RMSEp = 1.75–8.95 g/kg, RPD = 0.50–3.61; TN: RMSEp = 0.12–0.27 g/kg; RPD = 1.00–3.11) because of the higher number of characteristic bands. Prediction models based on the whole depth calibration (OM: RMSEp = 1.09–2.97 g/kg, RPD = 2.13–5.80; TN: RMSEp = 0.08–0.19 g/kg, RPD = 1.86–4.39) outperformed those based on the shallow layers (OM: RMSEp = 1.07–8.95 g/kg, RPD = 0.50–3.93; TN: RMSEp = 0.11–0.27 g/kg, RPD = 1.00–2.24) because the soil sample data of the whole depth had a larger and more representative sample size and a wider distribution. However, prediction models based on the whole depth calibration might provide lower accuracy in some shallow layers. Accordingly, it is suggested that the methods pertaining to soil property prediction based on the spectral library should be considered in future studies for an optimal approach to predicting soil properties at specific depths. This study verified the superiority of MIR for soil property prediction at specific depths and confirmed the advantage of modeling with the whole depth calibration, pointing out a possible optimal approach and providing a reference for predicting soil properties at specific depths.