In recent years, with increasing global demand for oil and gas resources and continuous decline in conventional oil and gas production, the global development focus has shifted from conventional medium to high permeability reservoirs to low permeability and tight oil reservoirs. As a result, nanoparticles (NPs) have found a promising role in enhanced oil recovery as potential improved oil recovery agents in low permeability. Despite many experiments that have proved that nanoparticles can be adsorbed on the rock surface in a macroscopic perspective, the adsorption mechanism and the effects of molecular structure on the adsorption behavior of nanoparticles on rock surfaces remain scarce. Here, the fundamental phenomena involved in hydrophobic nanoparticles adsorption on rock surface and the effect of mineral composition on adsorption mechanism were elucidated by the analysis of molecular dynamics simulation. The simulation results show that water molecules could form two adsorption layers on both quartz and kaolinite surfaces. Hydration layer thickness of kaolinite is greater than that of the quartz surface. The solid/liquid interface hydration layer thickness of quartz–water system is approximately 0.71 nm, while the thickness of kaolinite–water system is approximately 0.75 nm. Furthermore, coulombic interactions are the main influencing factor for the stable adsorption of nanoparticles on the wall. Nanoparticles can only break through the first adsorption layer to absorb on the layer. Finally, wetting angle tests were conducted which indicated that SiO2 nanoparticles can be adsorbed on the surface and have a good wetting reversal effect. Our study highlights the adsorption mechanism of nanoparticles on a molecular level, which may help to promote the development of low permeability and tight oil reservoirs.