A non-local action functional for electrodynamics depending on the electric and magnetic fields, instead of potentials, has been proposed in the literature. In this work we elaborate and improve this proposal. We also use this formalism to confront the electric-magnetic duality symmetry of the electromagnetic field and the Aharonov-Bohm effect, two subtle aspects of electrodynamics that we examine in a novel way. We show how the former can be derived from the simple harmonic oscillator character of vacuum electrodynamics, while also demonstrating how the magnetic version of the latter naturally arises in an explicitly non-local manner.