We develop an open-system dynamical theory of the Casimir interaction between coherent atomic waves and a material surface. The system -the external atomic waves -disturbs the environment -the electromagnetic field and the atomic dipole degrees of freedom -in a non-local manner by leaving footprints on distinct paths of the atom interferometer. This induces a non-local dynamical phase depending simultaneously on two distinct paths, beyond usual atom-optics methods, and comparable to the local dynamical phase corrections. Non-local and local atomic phase coherences are thus equally important to capture the interplay between the external atomic motion and the Casimir interaction. Such dynamical phases are obtained for finite-width wavepackets by developing a diagrammatic expansion of the disturbed environment quantum state.