In the past 50 years, the application of synthetic nitrogen (N) fertilizer to farmland resulted in a dramatic increase in crop yields but with considerable negative impacts on the environment. New solutions are therefore needed to simultaneously increase yields while maintaining, or preferably decreasing, applied N to maximize the nitrogen use efficiency (NUE) of crops. In this review, we outline the definition of NUE, the selection and development of NUE crops, and the factors that interact with NUE. In particular, we emphasize the challenges of developing crop plants with enhanced NUE, using more classical genetic approaches based on utilizing existing allelic variation for NUE traits. The challenges of phenotyping, mapping quantitative trait loci (QTLs), and selecting candidate genes for NUE improvement are described. In addition, we highlight the importance of different factors that lead to changes in the NUE components of nitrogen uptake efficiency (NUpE) and nitrogen utilization efficiency (NUtE).