Material characterization has been proved to be the most intuitive approach to understand the chemical composition, structure, and microstructure of materials, which is the basis of material design. One of the most important steps in material design is to extract the characteristics from an image, and find their associations with the material structure and properties. Therefore, in recent years, with the rapid development of machine vision algorithms, characterization images have attracted attention in the field of material characterization. Researchers use computer vision algorithms, such as image denoising and enhancement, to preprocess the representation image, image segmentation and classification to detect and separate each microstructure from the characterization image, and quantitatively analyze the properties of materials. Herein, the application of computer vision algorithms in material image representation is summarized and discussed. The latest and valuable views for experts and scholars in both computer vision and material grounds are presented. Thus, this review provides guidance for material exploration and promotes the developments of artificial intelligence in the field of materials.