The study of the (logarithm of the) fidelity i.e., of the overlap amplitude, between ground states of Hamiltonians corresponding to different coupling constants, provides a valuable insight on critical phenomena. When the parameters are infinitesimally close, it is known that the leading term behaves as O(L α ) (L system size) where α is equal to the spatial dimension d for gapped systems, and otherwise depends on the critical exponents. Here we show that when parameters are changed along a critical manifold, a sub-leading O(1) term can appear. This term, somewhat similar to the topological entanglement entropy, depends only on the system's universality class and encodes non-trivial information about the topology of the system. We relate it to universal g factors and partition functions of (boundary) conformal field theory in d = 1 and d = 2 dimensions. Numerical checks are presented on the simple example of the XXZ chain.