In the present paper we investigate the influence of the retarded access by a time-varying delay on the dynamics of delay systems. We show that there are two universality classes of delays, which lead to fundamental differences in dynamical quantities such as the Lyapunov spectrum. Therefore we introduce an operator theoretic framework, where the solution operator of the delay system is decomposed into the Koopman operator describing the delay access and an operator similar to the solution operator known from systems with constant delay. The Koopman operator corresponds to an iterated map, called access map, which is defined by the iteration of the delayed argument of the delay equation. The dynamics of this one-dimensional iterated map determines the universality classes of the infinite-dimensional state dynamics governed by the delay differential equation. In this way, we connect the theory of time-delay systems with the theory of circle maps and the framework of the Koopman operator. In the present paper we extend our previous work [Otto, Müller, and Radons, Phys. Rev. Lett. 118, 044104 (2017)], by elaborating the mathematical details and presenting further results also on the Lyapunov vectors.