Mesenchymal stem/progenitor cells (MSCs) have a multi-differentiation potential into specialized cell types, with remarkable regenerative and therapeutic results. Several factors could trigger the differentiation of MSCs into specific lineages, among them the biophysical and chemical characteristics of the extracellular matrix (ECM), including its stiffness, composition, topography, and mechanical properties. MSCs can sense and assess the stiffness of extracellular substrates through the process of mechanotransduction. Through this process, the extracellular matrix can govern and direct MSCs’ lineage commitment through complex intracellular pathways. Hence, various biomimetic natural and synthetic polymeric matrices of tunable stiffness were developed and further investigated to mimic the MSCs’ native tissues. Customizing scaffold materials to mimic cells’ natural environment is of utmost importance during the process of tissue engineering. This review aims to highlight the regulatory role of matrix stiffness in directing the osteogenic differentiation of MSCs, addressing how MSCs sense and respond to their ECM, in addition to listing different polymeric biomaterials and methods used to alter their stiffness to dictate MSCs’ differentiation towards the osteogenic lineage.