Multispectral LiDAR is an emerging active remote sensing technique that combines distance and spectroscopy measurements on light reflected from the surface at the respective measurement point. It is known that the reflectance spectrum can be used for material classification. However, the spectrum also depends on other surface parameters, particularly surface roughness. Herein, we propose an extension of multispectral to polarimetric multispectral LiDAR and introduce polarized and unpolarized reflectance spectra as additional features for classifying materials and roughness. We demonstrate the feasibility and the benefit using a bench-top prototype instrument which allows acquiring standard, polarized and unpolarized reflectance spectra, in addition to distance, in 33 spectral channels with 10 nm bandwidth between 580 and 900 nm. We analyze and interpret the raw spectra obtained from measurements on test specimens consisting of five different materials (PE, PVC, PP, sandstone, limestone) with two different levels of surface roughness. Using a linear support vector machine (SVM) we demonstrate the potential of the different features for independent material and roughness classification. The results indicate that the unpolarized reflectance spectrum increases the material classification accuracy by 50% as compared to a standard spectrum, and that the polarized spectrum actually allows classifying roughness. We interpret the results as a strong indication that multispectral polarimetric LiDAR enables deriving practically relevant additional information on surfaces with high spatial resolution through remote sensing.