In this study, the authors explore the opportunities, application areas and challenges involving the use of imaging spectroscopy as a means for scene understanding. This is important, since scene analysis in the scope of imaging spectroscopy involves the ability to robustly encode material properties, object composition and concentrations of primordial components in the scene. The combination of spatial and compositional information opens-up a vast number of application possibilities. For instance, spectroscopic scene analysis can enable advanced capabilities for surveillance by permitting objects to be tracked based on material properties. In computational photography, images may be enhanced taking into account each specific material type in the scene. For food security, health and precision agriculture it can be the basis for the development of diagnostic and surveying tools which can detect pests before symptoms are apparent to the naked eye. This combination of a broad domain of application with the use of key technologies makes the use of imaging spectroscopy a worthwhile opportunity for researchers in the areas of computer vision and pattern recognition.