Frequency conversion (FC) is an enabling process in many quantum information protocols. Recently, it has been observed that upconversion efficiencies in single-photon, mode-selective FC are limited to around 80%. In this letter we argue that these limits can be understood as time-ordering corrections (TOCs) that modify the joint conversion amplitude of the process. Furthermore we show, using a simple scaling argument, that recently proposed cascaded FC protocols that overcome the aforementioned limitations act as "attenuators" of the TOCs. This observation allows us to argue that very similar cascaded architectures can be used to attenuate TOCs in photon generation via spontaneous parametric down-conversion. Finally, by using the Magnus expansion, we argue that the TOCs, which are usually considered detrimental for FC efficiency, can also be used to increase the efficiency of conversion in partially mode selective FC.Nonlinear photonic materials provide some of the most advanced platforms for manipulating the frequency and spectral profile of photons. This manipulation is typically achieved using a process known as frequency conversion (FC) [1,2]. In the version of FC we consider, two photons (one of them typically coming from a bright classical field) are fused into another photon with a higher energy by the process of sum frequency generation. FC has several important applications, including photon detection [3], and the establishment of compatibility between sources and quantum memories [4]. FC can also be used to modify in a controlled manner the properties of weak signals or single photons, a useful component of several quantum information processing protocols that harness the infinite dimensional Hilbert space structure of the frequency degree of freedom of photons [5][6][7][8]. A set of orthogonal frequency amplitude functions for the photon(s) (henceforth referred as "modes") provides a natural basis in which to encode information in this Hilbert space [5], and FC provides a natural way to do controlled operations in this Hilbert space [6]. For all applications of FC it is important to have conversion efficiency near unity, and for controlled operations it is also important to have mode selectivity. In the limit of very short crystals, or equivalently very long pulses (effectively CW fields) it has been shown that 100% FC is achievable [4,9]. Nevertheless in these limits most mode selectivity is lost. In this letter, we examine limitations to highly efficient mode selective FC. We show below that these limitations are due to time-ordering corrections (TOCs) that appear because the interaction picture Hamiltonian that describes the χ 2 (or χ 3 ) interaction between the different fields does not commute with itself at different times. Our study allows us to separate very cleanly the "ideal" operation of an FC device from the "undesirable" effects of time ordering. Thus, we can explicitly write the operation of an FC gate, in a language very close to the one used in quantum information, as a unitary operati...