Aiming at the problems of high dimensional features, poor viewpoint robustness and long retrieval time of existing algorithms in the image retrieval system, this paper presents a new image retrieval algorithm by integrating image color information and surface geometry principal curvatures information. In the proposed method, the color image is first quantized and counted to obtain its color histogram. Simultaneously, the Hessian matrix is used to extract the texture information and the joint histogram of oriented gradient with mix-sampling and multi-scale is constructed. And then, the obtained color histogram and histogram of oriented gradient are fused to obtain the final joint histogram. Experiments are performed on public datasets, and comparison and analysis with representative algorithms based on a single visual feature or set of visual features to verify the performance of our algorithm. The experimental results show that the proposed method has the advantages of low dimensionality, fastness, strong viewpoint robustness and high precision, and can realize image retrieval efficiently.