In brain magnetic resonance (MR) images, image quality is often degraded due to the influence of noise and outliers, which brings some difficulties for doctors to segment and extract brain tissue accurately. In this paper, a modified robust fuzzy c-means (MRFCM) algorithm for brain MR image segmentation is proposed. According to the gray level information of the pixels in the local neighborhood, the deviation values of each adjacent pixel are calculated in kernel space based on their median value, and the normalized adaptive weighted measure of each pixel is obtained. Both impulse noise and Gaussian noise in the image can be effectively suppressed, and the detail and edge information of the brain MR image can be better preserved. At the same time, the gray histogram is used to replace single pixel during the clustering process. The results of segmentation of MRFCM are compared with the state-of-the-art algorithms based on fuzzy clustering, and the proposed algorithm has the stronger anti-noise property, better robustness to various noises and higher segmentation accuracy.