Nanoparticle vermiculite (VMT) clay was prepared by treatment with hydrochloric acid. Styrene‐butadiene rubber (SBR) nanocomposites were prepared by mixing different contents (2.5, 5, 7.5, and 10 phr) of untreated (VMT) and acid‐treated (DVMT) vermiculite clay, respectively. In addition, different contents (3, 7, and 10 phr) of maleic anhydride (MA) as compatibilizer were mixed via direct melt compounding in internal mixer. The effect of gamma irradiation, VMT clay, and MA contents on the mechanical properties was studied. The acid‐treated VMT clay was characterized by x‐ray diffraction (XRD), transmission electron microscopy (TEM), and Fourier transform infrared (FT‐IR) spectroscopy. Meanwhile, the SBR/VMT composites, SBR/DVMT, and SBR/DVMT/MA nanocomposites were characterized via crosslinking density and tensile mechanical testing and FT‐IR spectroscopic analysis. The results indicated that good yield of nanoparticle vermiculite was achieved when the acid treatment was carried out for 120 h. In addition, the results showed that the presence of DVMT clay improved the chemical bonding in the SBR nanocomposites and hence their mechanical properties. The highest improvement was obtained when the contents of DVMT clay, MA, and irradiation dose were 10 phr, 3 phr, and 100 kGy, respectively. POLYM. ENG. SCI., 59:355–364, 2019. © 2018 Society of Plastics Engineers