To capture color and near-infrared images simultaneously, a multi-spectral filter array(MSFA) sensor is used. This is because an NIR band gives additional invisible information to human eyes to see subject under extremely low light level. However, because lenses have different refractive indices for different wavelengths, lenses may fail to focus widely different rays to the same convergence point. This is why a chromatic aberration(CA) problem occurs and images are degraded. In this paper, the image restoration algorithm for an MSFA image, which removes the CA problem, is presented. The obtained MSFA image is filtered by the estimated low-pass kernel to generate a base image. This base image is used to remove CA problem in multi-spectral(MS) images. By modeling the image degradation process and by using the least squares approach of the difference between the high-frequencies of the base and MS images, the desired high-resolution MS images are reconstructed. The experimental results show that the proposed algorithm performs well in estimating the high-quality MS images and reducing the chromatic aberration problem.