For two-dimensional holographic CFTs, we demonstrate the role of Berry phases for relating the non-factorization of the Hilbert space to the presence of wormholes. The wormholes are characterized by a non-exact symplectic form that gives rise to the Berry phase. For wormholes connecting two spacelike regions in gravitational spacetimes, we find that the non-exactness is linked to a variable appearing in the phase space of the boundary CFT. This variable corresponds to a loop integral in the bulk. Through this loop integral, non-factorization becomes apparent in the dual entangled CFTs. Furthermore, we classify Berry phases in holographic CFTs based on the type of dual bulk diffeomorphism involved. We distinguish between Virasoro, gauge and modular Berry phases, each corresponding to a spacetime wormhole geometry in the bulk. Using kinematic space, we extend a relation between the modular Hamiltonian and the Berry curvature to the finite temperature case. We find that the Berry curvature, given by the Crofton form, characterizes the topological transition of the entanglement entropy in presence of a black hole.