Due to the impossibility of obtaining intact standard experimental samples, it is difficult to test the mechanical properties of soft and broken coal and rock obtained from deep coal mines. So, an advanced experimental technology based on a small sample volume, nanoindentation technology, was introduced and used to measure the mechanical parameters of them. By using the averaging method, the hardness of shale, mudstone and coal are 1191.90 MPa, 674.95 MPa and 424.30 MPa, respectively; their elastic moduli are 20.39 GPa, 11.72 GPa and 5.47 GPa; and their fracture toughness were 1.66 MPa·m0.5, 1.28 MPa·m0.5 and 0.77 MPa·m0.5. These three mechanical parameters were used to quantify and map the heterogeneous properties of coal and rock for convenience and accuracy. For example, the inter quartile range (IQR) of the hardness of shale, mudstone, and coal are 1502.10 MPa, 1016.20 MPa and 54.64 MPa, respectively, meaning that coal has the best homogeneity among them. Nanoindentation technology provides researchers with a convenient method to conduct mechanical experiments at the microscale.