The concentration of rare earth elements (REEs) in geological materials including SLRS-6 (natural water certified reference material) and JB1b, JA1, and JG2 (Standard Rock Materials of Geological Survey of Japan) can be used as a tracer to characterize various geochemical processes in earth systems. Particularly, accurate and precise determination of rare earth element concentration in natural waters is difficult due to their extremely low concentration and the interference of polyatomic oxides. In this study, we developed a method for accurate and precise determination of the REE (particularly heavy rare earth elements) concentrations in geological materials including natural waters using a multi-collector inductively coupled plasma mass spectrometer (MC-ICP-MS) and group separation by 2-hydroxyisobutyric acid (HIBA). The REEs were separated into light rare earth elements (LREEs, La–Ce–Pr–Nd), middle rare earth elements (MREEs, Sm–Eu–Gd–Tb), and heavy rare earth elements (HREEs, Dy–Ho–Er–Tm–Yb–Lu) by a cation-exchange column (AG50W-X8 200–400 mesh) using HIBA. The recovery rates of each REE in the natural water sample exceeded 98%, whereas the recovery rates of each REE in rock materials exceeded 95% except for HREEs. The method developed in this study can accurately measure the REE concentrations (particularly HREE) in geological materials without polyatomic oxide interference during the REE analysis by using the MC-ICP-MS and, thus, can correctly interpret the geochemical implications of REEs in geological systems. The determination of the Sr concentrations and Sr isotopic ratios of SLRS-6 CRM and JB1b, JA1, and JG2 SRMs is also reported, and they are shown to be in good agreement with the recommended values.