Quadratic programming over the (special) orthogonal group encompasses a broad class of optimization problems such as group synchronization, point-set registration, and simultaneous localization and mapping. Such problems are instances of the little noncommutative Grothendieck problem (LNCG), a natural generalization of quadratic combinatorial optimization where, instead of binary decision variables, one optimizes over orthogonal matrices. In this work, we establish an embedding of this class of LNCG problems over the orthogonal group onto a quantum Hamiltonian. This embedding is accomplished by identifying orthogonal matrices with their double cover (Pin and Spin group) elements, which we represent as quantum states. We connect this construction to the theory of free fermions, which provides a physical interpretation of the derived LNCG Hamiltonian as a two-body interacting-fermion model due to the quadratic nature of the problem. Determining extremal states of this Hamiltonian provides an outer approximation to the original problem, analogous to classical relaxations of the problem via semidefinite programming. When optimizing over the special orthogonal group, our quantum relaxation naturally obeys additional, powerful constraints based on the convex hull of rotation matrices. The classical size of this convex-hull representation is exponential in matrix dimension, whereas the quantum representation requires only a linear number of qubits. Finally, to project the relaxed solution into the feasible space, we employ rounding procedures which return orthogonal matrices from appropriate measurements of the quantum state. Through numerical experiments we provide evidence that this quantum relaxation can produce high-quality approximations.