Secondary ion mass spectrometry (SIMS) has a wide range of applications in Earth Science research, thanks to its high precision and sensitivity, and its capacity in direct insitu micromeasurement. The technique is operated in ultra‐high vacuum (UHV) conditions, especially for the measurement of volatiles such as hydrogen, or the water content in nominally anhydrous minerals (NAMs). To minimize the water background and obtain accurate and precise water contents in NAMs (eg, olivine) critical parameters such as presputtering time, field aperture (FA), dynamic transfer on/off, and primary beam current intensity were investigated for a CAMECA IMS 1280‐HR system. When the chamber vacuum reaches approximately 2 × 10−9 mbar, we set the DTOS OFF, raster size to 50 μm and primary beam current to 5 nA, and used 2000 μm FA and 170‐second presputtering time. Consequently, an approximately 1.2 ppmw water background and 3.6 ppmw limit of detection (LOD) were yielded, from analyzing the San Carlos olivine. Meanwhile, the water content and homogeneity of a range of olivine minerals were characterized for potential use as reference materials for SIMS water content measurement. Olivine water content calibration curve was also established by comparing the Fourier transform infrared (FTIR) results with the SIMS‐measured 16O1H−/16O− ratios. Accuracy and precision of water content measurement were estimated to be better than approximately 10% in this study.