The optimization of timber classification by grades and defect detection plays an important role in the production of timbers. Traditionally, a timber is manual cut by a worker according to his experience. Defect detection and classification of a timber are with great subjectivity. Meanwhile, the action is not safe enough. In this case, an automatic optimizing cross-cut saw to finish these tasks of timber classification by grades and defect detection is built significantly. Related algorithm and detailed procedure for optimizing cross-cut saw are proposed in this paper. Additionally, a vision system is used to capture images of a timber. Captured images are analyzed and processed. First, defects in these images are detected. Then the serviceable part (defect-free) of a timber can be determined. Based on the pretrained network, the timber can be classified. As the homography matrix has been known, the physical position can be confirmed. In our proposed system, the cutting list is transmitted from the industrial control computer to a motion control system, then the timber can be cut according to the cutting list automatically. In this paper, related algorithms and detailed procedure are given. Moreover, a new optimizing cross-cut saw is built. Experiments show that the processing time for each image is about 0.026s and the minimum mean average precision is 94.15%. In this case, it can make the optimizing cross-cut saw efficient, laborsaving and safe. Furthermore, related algorithms are suitable to improve a traditional automatic optimizing cross-cut saw.