Using the worldline quantum field theory (WQFT) formalism for classical scattering, we study the deflection of light by a heavy massive spinless/spinning object. WQFT requires the use of the worldline dressed propagator of a photon in a gravitational background, which we construct from first principles. The action required to set up the worldline path integral is constructed using auxiliary variables, which describe dynamically the spin degrees of freedom of the photon and take care of path ordering. We test the fully regulated path integral by recovering the photon-photon-graviton vertex. With the dressed propagator at hand, we follow the WQFT procedure by setting up the partition function and deriving the Feynman rules which can be used to evaluate it perturbatively. These rules depend on the auxiliary variables. The latter ultimately do not contribute in the geometric-optics regime, which realizes the equivalence between the scattering of a photon and a massive scalar with that of a massless and a massive scalar. Then, the calculation of the eikonal phase and the deflection angle simplifies considerably. Using the eikonal phase defined in terms of the partition function, we calculate explicitly the deflection angle at NLO in the spinless case, and at LO in the spinning case up to quadratic order in spin.