Photonic time-varying systems have attracted significant attention owing to their rich physics and potential opportunities for new and enhanced functionalities. In this context, the duality of space and time in wave physics has been particularly fruitful to uncover interesting physical effects in the temporal domain, such as reflection/refraction at temporal interfaces and momentum-bandgaps in time crystals. However, the characteristics of the temporal/frequency dimension, particularly its relation to causality and energy conservation (
ℏ
ω
is energy, whereas
ℏ
k
is momentum), create challenges and constraints that are unique to time-varying systems and are not present in their spatially varying counterparts. Here, we overview two key physical aspects of time-varying photonics that have only received marginal attention so far, namely temporal dispersion and external power requirements, and explore their implications. We discuss how temporal dispersion, an inherent property of any physical causal material, makes the fields evolve continuously at sharp temporal interfaces and may limit the strength of fast temporal modulations and of various resulting effects. Furthermore, we show that changing the refractive index in time always involves large amounts of energy. We derive power requirements to observe a time-crystal response in one of the most popular material platforms in time-varying photonics, i.e., transparent conducting oxides, and we argue that these effects are almost always obscured by less exotic nonlinear phenomena. These observations and findings shed light on the physics and constraints of time-varying photonics, and may guide the design and implementation of future time-modulated photonic systems.