When noninteracting fermions are confined in a D-dimensional region of volume O(L D ) and subjected to a continuous (or piecewise continuous) potential V which decays sufficiently fast with distance, in the thermodynamic limit, the ground state energy of the system does not depend on V , but only on the imposed boundary conditions. Here, we review this almost unknown result from several perspectives and derive a proof for radially symmetric potentials valid in D = 2 and D = 3 dimensions, generalizing an existing approach to the case D = 1. We find that this universality property holds under a quite mild condition on V and extends to thermal states. Moreover, it leads to an interesting analogy between Anderson's orthogonality catastrophe and first-order quantum phase transitions.