Nitrogen (N) fertilizer plays an important role in wheat yield, but N application rates vary greatly, and there is a lack of data to quantify the residual effects of N fertilization on soil N availability. A 17-yr experiment was conducted in a semiarid area of the Loess Plateau of China to assess the effects of N fertilization on spring wheat (Triticum aestivum L.) grain yield, N uptake, N utilization efficiency, and residual soil nitrate. Treatments included a non-N-fertilized control and annual application of 52.5, 105.0, 157.5, and 210.0 kg N ha−1 in the first two years (2003 and 2004). In the third year (2005), the four main plots with N fertilizer application were split. In one subplot, N fertilization was continued as mentioned previously, while in the other subplot, N fertilization was stopped. The concentration of NO3-N in the 0–110 cm depth soil layers was significantly affected by N application, with higher N rates associated with greater soil NO3-N concentration. With the annual application of N over 17 years, residual soil NO3-N concentration in the 100–200 cm soil layer in the last study year was significantly greater than that in the non-N-fertilized control and was increased with rate of N application. There was a significant positive relationship of soil NO3-N in the 0–50 cm and 50–110 cm soil layers at wheat sowing with wheat grain N content and yield. Wheat grain yield in the third year (2005) was significantly, i.e., 22.57–59.53%, greater than the unfertilized treatment after the N application was stopped. Nitrogen use efficiency decreased in response to each increment of added N fertilizer, and was directly related to N harvest index and grain yield. Therefore, greater utilization of residual soil N through appropriate N fertilizer rates could enhance nitrogen use efficiency while reducing the cost of crop production and risk of N losses to the environment. For these concerns, optimum N fertilizer application rate for spring wheat in semiarid Loess Plateau is about 105 kg N ha−1, which is below the threshold value of 170 kg N ha−1 per year as defined by most EU countries.