Describing a particle in an external electromagnetic field is a basic task of quantum mechanics. The standard scheme for this is known as "minimal coupling", and consists of replacing the momentum operators in the Hamiltonian by modified ones with an added vector potential. In lattice systems it is not so clear how to do this, because there is no continuous translation symmetry, and hence there are no momenta. Moreover, when time is also discrete, as in quantum walk systems, there is no Hamiltonian, only a unitary step operator. We present a unified framework of gauge theory for such discrete systems, keeping a close analogy to the continuum case. In particular, we show how to implement minimal coupling in a way that automatically guarantees unitary dynamics. The scheme works in any lattice dimension, for any number of internal degree of freedom, for walks that allow jumps to a finite neighborhood rather than to nearest neighbours, is naturally gauge invariant, and prepares possible extensions to non-abelian gauge groups.