A magnetic polymer-based nanocomposite was fabricated by the modification of an Fe3O4/SiO2 magnetic composite with polypyrrole (PPy) via co-precipitation polymerization to form PPy/Fe3O4/SiO2 for the removal of Congo red dye (CR) and hexavalent chromium Cr(VI) ions from water. The nanocomposite was characterized using various techniques including X-ray diffraction (XRD), Fourier transform infrared spectroscopy (FTIR), scanning electron microscope (SEM), vibration sample magnetometer, and thermogravimetric analysis (TGA). The results confirm the successful fabrication of the nanocomposite in the size of nanometers. The effect of different conditions such as the contact time, adsorbent dosage, solution pH, and initial concentration on the adsorption process was investigated. The adsorption isotherm suggested monolayer adsorption of both contaminants over the PPy/Fe3O4/SiO2 nanocomposite following a Langmuir isotherm, with maximum adsorption of 361 and 298 mg.g−1 for CR dye and Cr(VI), respectively. Furthermore, the effect of water type on the adsorption process was examined, indicating the applicability of the PPy/Fe3O4/SiO2 nanocomposite for real sample treatment. Interestingly, the reusability of the nanocomposite for the removal of the studied contaminants was investigated with good results even after six successive cycles. All results make this nanocomposite a promising material for water treatment.