Recently, the family of rare-earth chalcohalides were proposed as candidate compounds to realize the Kitaev spin liquid (KSL) [Chin. Phys. Lett. 38 047502 (2021)]. In the present work, we firstly propose an effective spin Hamiltonian consistent with the symmetry group of the crystal structure. Then we apply classical Monte Carlo simulations to preliminarily study the model and establish a phase diagram. When approaching to the low temperature limit, several magnetic long range orders are observed, including the stripe, the zigzag, the antiferromagnetic (AFM), the ferromagnetic (FM), the incommensurate spiral (IS), the multi- Q , and the 120° ones. We further calculate the thermodynamic properties of the system, such as the temperature dependence of the magnetic susceptibility and the heat capacity. The ordering transition temperatures reflected in the two quantities agree with each other. For most interaction regions, the system is magnetically more susceptible in the ab-plane than in the c-direction. The stripe phase is special, where the susceptibility is fairly isotropic in the whole temperature region. These features provide useful information to understand the magnetic properties of related materials.

Rare-earth chalcohalide REChX (RE = rare earth; Ch = O, S, Se, Te; X = F, Cl, Br, I) is a newly reported family of Kitaev spin liquid candidates. The family offers a platform where a strong spin-orbit coupling meets a van der Waals layered and undistorted honeycomb spin lattice, which outputs highly anisotropic exchange couplings required by the Kitaev model. YbOCl is the first single crystal of the family we grew, with a size up to ∼15 mm. We have performed magnetization and high magnetic field electron spin resonance measurements from 2 to 300 K. We develop the mean-field scenario for the anisotropic spin system, with which we are able to well describe the experiments and reliably determine the fundamental parameters. The self-consistent simulations give the anisotropic spin-exchange interactions of J ± (∼ − 0.3 K) and J zz (∼1.6 K), and g factors of g ab (∼3.4) and g c (∼2.9). Based on the spin-exchange interactions, we employ the exact diagonalization method to work out the ground state phase diagram of YbOCl in terms of the off-diagonal exchange couplings. The phase diagram hosting rich magnetic phases including the spin-disordered one, sheds light on the novel magnetic properties of the family, particularly the Kitaev physics.