Computational modeling is a crucial approach in material-related research for discovering new materials with superior properties. However, the high design flexibility in materials, especially in the realm of metamaterials where the sub-wavelength structure provides an additional degree of freedom in design, poses a formidable computational cost in various real-world applications. With the advent of big data, deep learning brings revolutionary breakthroughs in many conventional machine learning and pattern recognition tasks such as image classification. The accompanied data-driven modeling paradigm also provides transformative methodology shift in materials science, from trial-and-error routine to intelligent material discovery and analysis. This review systematically summarize the application of deep learning in material science, based on a model selection perspective for both natural materials and metamaterials. The review aims to uncover the logic behind data-model relation with emphasis on suitable data structures for different scenarios in the material study and the corresponding problem-solving deep learning model architectures.