Fouling is a major obstacle and challenge in membrane‐based separation processes. Caused by the sophisticated interactions between foulant and membrane surface, fouling strongly depends on membrane surface chemistry and roughness. Current studies in the field have been largely focused on polymer membranes. Herein, a molecular dynamics simulation study is reported for fouling on alumina and graphene membrane surfaces during water treatment. For two foulants (sucralose and bisphenol A), it is found that the fouling on alumina surfaces is reduced with increasing surface roughness; however, the fouling on graphene surfaces is enhanced by roughness. It is unravelled that the foulant–surface interaction becomes weaker in the ridge region of a rough alumina surface, thus allowing foulant to leave the surface and reducing fouling. Nevertheless, such behavior is not observed on a rough graphene surface because of strong foulant–graphene interaction. In addition, with increasing roughness, the hydrogen bonds between water and alumina surface are found to increase in number as well as stability. This simulation study reveals that surface chemistry and roughness play a crucial role in membrane fouling, and the microscopic insights are useful for the design of new membranes toward high‐performance water treatment.