Galaxy clusters and groups are important cosmological probes and giant cosmic laboratories for studying galaxy evolution. Much effort has been devoted to understanding how and when baryonic matter cools at the centre of potential wells. However, a clear picture of the efficiency with which baryons are converted into stars is still missing. We present the K-band luminosity-halo mass relation, L K,500 − M 500,WL , for a subsample of 20 of the 100 brightest clusters in the XXL Survey observed with WIRCam at the Canada-France-Hawaii Telescope (CFHT). For the first time, we have measured this relation via weak-lensing analysis down to M 500,WL = 3.5 × 10 13 M . This allows us to investigate whether the slope of the L K − M relation is different for groups and clusters, as seen in other works. The clusters in our sample span a wide range in mass, M 500,WL = 0.35−12.10 × 10 14 M , at 0 < z < 0.6. The K-band luminosity scales as log 10 (L K,500 /10 12 L ) ∝ β log 10 (M 500,WL /10 14 M ) with β = 0.85 +0.35 −0.27 and an intrinsic scatter of σ ln L K |M = 0.37 +0.19 −0.17 . Combining our sample with some clusters in the Local Cluster Substructure Survey (LoCuSS) present in the literature, we obtain a slope of 1.05 +0.16 −0.14 and an intrinsic scatter of 0.14 +0.09 −0.07 . The flattening in the L K − M seen in previous works is not seen here and might be a result of a bias in the mass measurement due to assumptions on the dynamical state of the systems. We also study the richness-mass relation and find that group-sized halos have more galaxies per unit halo mass than massive clusters. However, the brightest cluster galaxy (BCG) in low-mass systems contributes a greater fraction to the total cluster light than BCGs do in massive clusters; the luminosity gap between the two brightest galaxies is more prominent for group-sized halos. This result is a natural outcome of the hierarchical growth of structures, where massive galaxies form and gain mass within low-mass groups and are ultimately accreted into more massive clusters to become either part of the BCG or one of the brighter galaxies.