A new bilateral frame rate up-conversion (FRUC) algorithm based on the bi-directional comparison of the texture complexity in the reference frame is proposed. In the proposed algorithm, the accuracy of the motion estimation (ME) process is determined by the comparison of the prediction size in the reference region of the bi-directional ME process. Base on this comparison, the adaptive refinement process is adjusted. In the experimental results, the proposed algorithm outperforms the conventional bilateral FRUC algorithm by 0.27 dB.