The presence of molecular mutations in colorectal cancer (CRC) is a decisive factor in selecting the most effective first-line therapy. However, molecular analysis is routinely performed only in a limited number of patients with remote metastases. We propose to use tissue stiffness as a marker of the presence of molecular mutations in CRC samples. For this purpose, we applied compression optical coherence elastography (C-OCE) to calculate stiffness values in regions corresponding to specific CRC morphological patterns (n = 54). In parallel to estimating stiffness, molecular analysis from the same zones was performed to establish their relationships. As a result, a high correlation between the presence of KRAS/NRAS/BRAF driver mutations and high stiffness values was revealed regardless of CRC morphological pattern type. Further, we proposed threshold stiffness values for label-free targeted detection of molecular alterations in CRC tissues: for KRAS, NRAS, or BRAF driver mutation—above 803 kPa (sensitivity—91%; specificity—80%; diagnostic accuracy—85%), and only for KRAS driver mutation—above 850 kPa (sensitivity—90%; specificity—88%; diagnostic accuracy—89%). To conclude, C-OCE estimation of tissue stiffness can be used as a clinical diagnostic tool for preliminary screening of genetic burden in CRC tissues.