We perform supercurrent and tunneling spectroscopy measurements on gate-tunable InAs/Al Josephson junctions (JJs) in an in-plane magnetic field, and report on phase shifts in the current-phase relation measured with respect to an absolute phase reference. The impact of orbital effects is investigated by studying multiple devices with different superconducting lead sizes. At low fields, we observe gate-dependent phase shifts of up to φ 0 = 0.5π which are consistent with a Zeeman field coupling to highlytransmissive Andreev bound states via Rashba spin-orbit interaction. A distinct phase shift emerges at larger fields, concomitant with a switching current minimum and the closing and reopening of the superconducting gap. These signatures of an induced phase transition, which might resemble a topological transition, scale with the superconducting lead size, demonstrating the crucial role of orbital effects. Our results elucidate the interplay of Zeeman, spin-orbit and orbital effects in InAs/Al JJs, giving