We investigate the augmentation of active imaging with passive polarimetric imaging for material classification. Experiments are conducted to obtain a multimodal dataset of lidar reflectivity and polarimetric thermal self-emission measurements against a diverse set of material types. Using the assumption that active lidar imaging can provide high-resolution threedimensional spatial information, a known surface orientation is utilized to enable higher fidelity classification. Machine learning is applied to the dataset of monostatic lidar unidirectional reflectivity and passive longwave infrared degree of linear polarization features for material classification. The hybrid sensor technique can classify materials with 91.1% accuracy even with measurement noise resulting in a signal-to-noise ratio of only 6 dB. The application of the proposed technique is applicable for the classification of hidden objects or could assist existing spatial-based object classification. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.