Roughness ubiquitously prevails in rock joints and controls the shear behaviours, permeability and damage characteristics of rock joints. A plethora of investigations have focused on the description of joint roughness; however, a detailed method for quantifying joint roughness and evaluating the shear strength has not yet been established. In this study, within the framework of fractal theory, an optical measurement scale was defined to depict the fractal characteristics of joint roughness, and a boundary measurement scale was used to identify first- and second-order asperities. A composite indicator $$\eta$$
η
, including the fractal roughness coefficient ($$R_{D}$$
R
D
), the coefficient describing the roughness order ($$\mu$$
μ
) and the anisotropy parameter ($$\lambda$$
λ
), was proposed to quantify the surface morphology, which takes the asperity distribution and roughness anisotropy into account. The relationship between $$\eta$$
η
and JRC was established, and the JRC–JCS criterion was further updated. Moreover, representative examples were given to show the steps required to quantify the morphology of rough joint surfaces with the new quantitative parameter. Direct shear tests were conducted to validate the effectiveness of the proposed method in describing joint roughness and estimating joint shear strength; the results indicate that it is appropriate to use $$\eta$$
η
to estimate the joint roughness and that the proposed shear strength criterion can appropriately predict the shear strength within an acceptable error.