The filtering method of image with salt & pepper noise is presented in this paper, which is based on discrete grey model (DGM). First, using four directional convolution operators detect noise point and noise-free point. In evidence, the threshold affects the performance of noise detection. We can determine a reasonable threshold using a genetic algorithm to minimize the mean square error of the filtered image in the paper. DGM is used to process this noise point, which uses a pixel grey value of eight points around the noise point of image to build up a discrete grey model, and the first forecasting value replaces the original pixel grey value of the noise point to realize the forecast filter, and noise-free points are not processed. Comparison experiments between the proposed algorithm and three other methods are carried out under the different noise level of image. Experimental results show that the proposed algorithm has better processing effect for images with salt & pepper noise, where the filter effect has been evaluated by using the mean square error and peak signal to noise ratio objectively.