Hyperspectral imagery has been widely used in real applications such as remote sensing, agriculture, surveillance, and geological analysis. Matching hyperspectral images is a challenge task due to the high dimensional nature of the data. The matching task becomes more difficult when images with different dimensions, such as a hyperspectral image and an RGB image, have to be matched. In this paper, we address this problem by investigating structured support vector machine to learn graph model for each type of image. The graph model incorporates both low-level features and stable correspondences within images. The inherent characteristics are depicted by using graph matching algorithm on weighted graph models. We validate the effectiveness of our method through experiments on matching hyperspectral images to RGB images, and hyperspectral images with different dimensions.