Using a novel variant of polarized light microscopy for high-contrast imaging and quantification of unstained histology slides, the current study assesses the prognostic potential of peri-tumoral collagenous stroma architecture in 32 human stage III colorectal cancer (CRC) patient samples. We analyze three distinct polarimetrically-derived images and their associated texture features, explore different unsupervised clustering algorithm models to group the data, and compare the resultant groupings with patient survival. The results demonstrate an appreciable total accuracy of ~ 78% with significant separation (p < 0.05) across all approaches for the binary classification of 5-year patient survival outcomes. Surviving patients preferentially belonged to Cluster 1 irrespective of model approach, suggesting similar stromal microstructural characteristics in this sub-population. The results suggest that polarimetrically-derived stromal biomarkers may possess prognostic value that could improve clinical management/treatment stratification in CRC patients.