Because of the low impurity removal rate and high damage rate of the potato combine harvester during the working process, a new type of impurity removal device for the potato combine harvester was designed. The device was mainly composed of retaining plates, impurity removal rollers, smooth rollers, and a control system. To determine the best working parameters of the device, the rotational speed of the rollers, the distance between two rollers, and the inclination angle of the device were used as the test factors. The impurity removal rate and damage rate were employed as the evaluation indexes. Then, the single factor test and the second rotation orthogonal regression test were performed. The optimum combination after parameter optimization was determined to be as follows: rotational speed of the rollers was 113.4 r/min, the distance between the two rollers was 97.5 mm and the inclination angle of the device was 10.4°. The corresponding impurity removal rate was 95.12%, and the damage rate was 0.54%. The device can better meet the requirements of impurity removal operations.