Image quality assessment that aims to evaluate the image quality automatically by a computational model plays a significant role in image processing systems. To meet the need of accuracy and effectiveness, in the proposed method, complementary features including histogram of oriented gradient, edge information, and color information are employed for joint representation of the image quality. Afterwards, the dissimilarities of the extracted features between the distorted and reference images are quantified. Finally, support vector regression is used for distortion indices fusion and objective quality mapping. Experimental results validate that the proposed method outperforms the state-of-the-art methods in terms of consistency with subjective perception and robustness across various databases and different distortion types.