We study the potential of dijet photoproduction measurements at a future electron-ion collider (EIC) to better constrain our present knowledge of the nuclear parton distribution functions. Based on theoretical calculations at next-to-leading order and approximate next-to-next-to-leading order of perturbative QCD, we establish the kinematic reaches for three different EIC designs, the size of the parton density function modifications for four different light and heavy nuclei from He-4 over C-12 and Fe-56 to Pb-208 with respect to the free proton, and the improvement of EIC measurements with respect to current determinations from deep-inelastic scattering and Drell-Yan data alone as well as when also considering data from existing hadron colliders.