Octylamine (OTA), 1-dodecanethiol (DDT), and tri-n-octylphosphine (TOP) capped magnetite nanoparticles were prepared by co-precipitation method. Powder X-ray diffraction patterns confirmed inverse spinel crystalline phases for the as-prepared iron oxide nanoparticles. Transmission electron microscopic micrographs showed iron oxide nanoparticles with mean particle sizes of 2.1 nm for Fe3O4-OTA, 5.0 nm for Fe3O4-DDT, and 4.4 nm for Fe3O4-TOP. The energy bandgap of the iron oxide nanoparticles ranges from 2.25 eV to 2.76 eV. The iron oxide nanoparticles were used as photocatalysts for the degradation of methylene blue with an efficiency of 55.5%, 58.3%, and 66.7% for Fe3O4-OTA, Fe3O4-DDT, and Fe3O4-TOP, respectively, while for methyl orange the degradation efficiencies were 63.8%, 47.7%, and 74.1%, respectively. The results showed that tri-n-octylphosphine capped iron oxide nanoparticles are the most efficient iron oxide nano-photocatalysts for the degradation of both dyes. Scavenger studies show that electrons (e−) and hydroxy radicals (•OH) contribute significantly to the photocatalytic degradation reaction of both methylene blue and methyl orange using Fe3O4-TOP nanoparticles. The influence of the dye solution’s pH on the photocatalytic reaction reveals that a pH of 10 is the optimum for methylene blue degradation, whereas a pH of 2 is best for methyl orange photocatalytic degradation using the as-prepared iron oxide nano-photocatalyst. Recyclability studies revealed that the iron oxide photocatalysts can be recycled three times without losing their photocatalytic activity.