An important problem in Quantum Field Theory (QFT) is to understand the structures of observables on spacetime manifolds of nontrivial topology. Such observables arise naturally when studying physical systems at finite temperature and/or finite volume and encode subtle properties of the underlying microscopic theory that are often obscure on the flat spacetime. Locality of the QFT implies that these observables can be constructed from more basic building blocks by cutting-and-gluing along a spatial slice, where a crucial ingredient is the Hilbert space on the spatial manifold. In Conformal Field Theory (CFT), thanks to the operator-state correspondence, we have a non-perturbative understanding of the Hilbert space on a spatial sphere. However it remains a challenge to consider more general spatial manifolds. Here we study CFTs in spacetime dimensions d > 2 on the spatial manifold T2 × ℝd−3 which is one of the simplest manifolds beyond the spherical topology. We focus on the ground state in this Hilbert space and analyze universal properties of the ground state energy, also commonly known as the Casimir energy, which is a nontrivial function of the complex structure moduli τ of the torus. The Casimir energy is subject to constraints from modular invariance on the torus which we spell out using PSL(2, ℤ) spectral theory. Moreover we derive a simple universal formula for the Casimir energy in the thin torus limit using the effective field theory (EFT) from Kaluza-Klein reduction of the CFT, with exponentially small corrections from worldline instantons. We illustrate our formula with explicit examples from well-known CFTs including the critical O(N) model in d = 3 and holographic CFTs in d ≥ 3.