With the rapid development of machine learning and document analysis techniques, the research on the character mosaic and paper scrap recovery based on simulated annealing algorithm is urgent needed. In this paper, we approach is proposed based on pieces of text line features, form line semi-auto stitching method, mainly introduced the fragments of text line direction, form line direction acquisition algorithm, and the debris semi-auto stitching algorithm based on boundary intersection distance, pieces of the algorithm does not depend on the geometric features, implementation is simple and reliability is better. Also analyses the algorithm of computing workload, analysis shows that for the majority of less than 200 of the total number of pieces of actual scraps of paper, joining together the computing workload within the scope of the permit. The experiment simulation proves the effectiveness and feasibility of our proposed method.