Calcareous sand is developed by the fracture of marine biological skeleton under the impact of seawater. Calcareous sand is not transported in the process of deposition. Therefore, calcareous sand retains the characteristics of marine biological skeleton, low strength, and porous. In order to study the effect of nanoclay and cement on the modification of calcareous sand, a series of tests were carried out on the modified cement calcareous sand (CCS) with different content of nanoclay. In this study, the triaxial mechanical properties and failure modes of nanoclay and cement composite modified calcareous sand (NCCS) were studied through the triaxial UU test. Then, SEM tests were carried out on CCS and NCCS samples, and the micromechanism of nanoclay and cement composite modified Nanhai calcareous sand was analyzed. The results showed that (1) the shear properties of CCS could be improved by adding nanoclay. The optimum admixture ratio of nanoclay was 8%, and its peak stress was 23%-39% higher than that of CCS. (2) The peak stress and strain of NCCS showed a linear correlation. (3) Compared with CCS, the internal friction angle and cohesion of NCCS were increased by 5.2% and 52%, respectively. (4) Nanoclay could improve the compactness and structure of cement calcareous sand, and the macroscopic performance is the improvement of peak stress and cohesion.