We present a computationally efficient approach to solve the time-dependent Kohn-Sham equations in real-time using higher-order finite-element spatial discretization, applicable to both pseudopotential and all-electron calculations. To this end, we develop an a priori mesh adaption technique, based on the semi-discrete (discrete in space but continuous in time) error estimate on the time-dependent Kohn-Sham orbitals, to construct an efficient finite-element discretization. Subsequently, we obtain the full-discrete error estimate to guide our choice of the time-step. We employ spectral finite-elements along with special reduced order quadrature to render the overlap matrix diagonal, thereby simplifying the inversion of the overlap matrix that features in the evaluation of the discrete time-evolution operator. We use the second-order Magnus operator as the time-evolution operator, wherein the action of the discrete Magnus operator, expressed as exponential of a matrix, on the Kohn-Sham orbitals is obtained efficiently through an adaptive Lanczos iteration. We observe close to optimal rates of convergence of the dipole moment with respect to spatial and temporal discretization, for both pseudopotential and all-electron calculations. We demonstrate a staggering 100-fold reduction in the computational time afforded by higher-order finite-elements over linear finite-elements, for both pseudopotential and all-electron calculations. Further, for similar level of accuracy, we obtain significant computational savings by our approach as compared to state-of-theart finite-difference methods. We also demonstrate the competence of higher-order finite-elements for all-electron benchmark systems. Lastly, we observe good parallel scalability of the proposed method on many hundreds of processors.