We demonstrate that a large class of one-dimensional quantum and classical exchange models can be described by the same type of graphs, namely, Cayley graphs of the permutation group. Their well-studied spectral properties allow us to derive crucial information about those models of fundamental importance in both classical and quantum physics, and to completely characterize their algebraic structure. Notably, we prove that the spectral gap can be obtained in polynomial computational time, which has strong implications in the context of adiabatic quantum computing with quantum spin chains. This quantity also characterizes the rate to stationarity of some important classical random processes such as interchange and exclusion processes. Reciprocally, we use results derived from the celebrated Bethe ansatz to obtain mathematical results about these graphs in the unweighted case. We also discuss extensions of this unifying framework to other systems, such as asymmetric exclusion processes-a paradigmatic model in nonequilibrium physics-or the more exotic non-Hermitian quantum systems.