Using heuristic method to automatically generate test cases is a research hotspot at present. Although its advantages are obvious, it is slightly insufficient in the selection of optimal individuals. In this paper, aiming at the problems existing in the evaluation and selection of the optimal individual at present, based on the comprehensive analysis of the characteristics of layer proximity and branch distance function, a test case evaluation algorithm combining layer proximity and branch distance function is proposed. The basic idea of this algorithm is to select the individuals with high proximity between the actual execution path and the target path, and then select the individuals with the smallest branch distance in these individuals, so as to obtain the individuals with the best navigation ability. Experiments show that the proposed algorithm can quickly find test cases, especially for the test case generation of multi-layer nested programs.