Recent advances in high level finite element systems have allowed for the symbolic representation of discretisations and their efficient automated implementation as model source code. This allows for the extremely compact implementation of complex non-linear models in a handful of lines of high level code. In this work we extend the high level finite element FEniCS system to introduce an abstract representation of the temporal discretisation: this enables the similarly rapid development of transient finite element models. Efficiency is achieved via aggressive optimisations that exploit the temporal structure, such as automated pre-assembly and caching of forms, and the robust re-use of matrix factorisations and preconditioner data. The resulting models are as fast or faster than hand-optimised finite element codes. The high level representation of the system remains extremely compact and easily manipulated. This structure is exploited to derive the associated discrete adjoint model automatically, with the adjoint model inheriting the performance advantages of the forward model. Combined, this provides a system for the rapid development of efficient transient models, together with their discrete adjoints.