We discuss the use of time and frequency degrees of freedom of single photons in quantum optics. Such a degree of freedom is generally discretized into modes for experimental reasons, but it is not a physical requirement. The origin of the quantumness of the time and frequency variables can be explained because of the non-commutativity of time and frequency operators -which can be defined properly-when restricted to the one photon per mode subspace. As a consequence, We will show that frequency and time operators can be used to define a universal set of gates in this particular subspace and provide an experimental implementation of such a universal set of gates.