Polarimetric and hyperspectral imaging are two of the most frequently used remote sensing modalities. While extensive work has been done in both fields independently, relatively little work has been done using both in conjunction with one another. Combining these two common remote sensing techniques, we hope to estimate index of refraction, without a priori knoweledge of local weather conditions or object surface temperature. In general, this is an underdetermined problem, but modeling the spectral behavior of the index of refraction reduces the number of parameters needed to describe the index of refraction, and thus the reflectively. This allows additional scene parameters needed to describe the radiance signature from a target to be simulataneously solved for. The method uses spectrally resolved S 0 and S 1 radiance measurements, taken using an IFTS with a linear polarizer mounted to the front, to simultaneously solve for a materials index of refraction, surface temperature, and downwelling radiance. Measurements at multiple angles relative to the surface normal can also be taken to provide further constraining information in the fit. Results on both simulated and measured data are presented showing that this technique is largely robust to changes in object temperature.