Image quality assessment (IQA) model is designed to measure the image quality in consistent with subjective ratings by computational models. In this research, a reliable full reference color IQA model is proposed by combining the Visual saliency with Color appearance (VC) similarity, gradient similarity and chrominance similarity. Two new color appearance indices, vividness and depth, are selected to build the visual saliency similarity map. The structure and chrominance features are characterized by different channels of chosen color space. VC map plays two roles in the proposed model. One is utilized as feature to compute the local quality of distorted image, and the other is as a weight part to reflect the importance of local domain. The novel model is called visual saliency with color appearance and gradient similarity (VCGS). To quantify the specific parameters of VCGS, some experiments are conducted based on the statistical correlation indices. Massive experiments are performed on the publicly available benchmark single and multiple distortion databases, and the commonly evaluation criteria results prove that VCGS works with higher consistency with the subjective evaluations than the other state-of-the-art IQA models for the prediction accuracy. Besides, VCGS maintain a moderate computational complexity.