Surface roughening of the substrates before bonding plays a significant effect on improving the mechanical performance of the adhesively bonded joints, which are prevalently used for light-weighting vehicle bodies. In this study, the influence of surface roughening on the lap shearing strength and failure behavior of adhesively bonded aluminum sheet joints was investigated. Sandpaper grinding was employed for surface roughening, methods such as tensile testing microstructure observation, etc., was employed for evaluating the performance of the joints. The results showed that the lap shearing strength of adhesively bonded joints increased and then decreased with the surface roughness of the aluminum substrate. The maximum shearing strength of the joint bonded with grinded substrates was 30.4 MPa which was improved by 57.5% compared to that produced with un-grinded substrates. However, over roughening is harmful. When the surface roughness was too large, the failure mode of the joint turned from the mixed failure mode to interfacial failure mode, which decreased the strength of the joints. Related mechanisms were demonstrated. When the substrate surface was coarsened, the bonding area and the wettability of the adhesive on the surface were both increased, which promoted the beneficial mechanical interlock effect between the adhesive and the substrate. However, when the substrate surface was over roughened, defects such as voids, insufficient infiltrating of the adhesive, etc., were induced, which apparently increased the proportion of the interfacial failure area.