In this paper, we use an implicit two-derivative deferred correction time discretization approach and combine it with a spatial discretization of the discontinuous Galerkin spectral element method to solve (non-)linear PDEs. The resulting numerical method is high order accurate in space and time. As the novel scheme handles two time derivatives, the spatial operator for both derivatives has to be defined. This results in an extended system matrix of the scheme. We analyze this matrix regarding possible simplifications and an efficient way to solve the arising (non-)linear system of equations. It is shown how a carefully designed preconditioner and a matrix-free approach allow for an efficient implementation and application of the novel scheme. For both, linear advection and the compressible Euler equations, up to eighth order of accuracy in time is shown. Finally, it is illustrated how the method can be used to approximate solutions to the compressible Navier-Stokes equations.