Various sources of noise limit the performance of quantum computers by altering qubit states in an uncontrolled manner throughout computations and reducing their coherence time. In quantum annealers, this noise introduces additional fluctuations to the parameters defining the original problem Hamiltonian, such that they find the ground states of problems perturbed from those originally programmed. Here we describe a method to benchmark the amount of noise affecting the programmed Hamiltonian of a quantum annealer. We show that a sequence of degenerate runs with the coefficients of the programmed Hamiltonian set to zero leads to an estimate of the noise spectral density affecting Hamiltonian parameters "in situ" during the quantum annealing protocol. The method is demonstrated in D-Wave's lower noise 2000 qubit device (DW_2000Q_6) and in its recently released 5000 qubit device (Advantage_system1.1). Our benchmarking of DW_2000Q_6 shows Hamiltonian noise dominated by the 1/f 0.7 frequency dependence characteristic of flux noise intrinsic to the materials forming flux qubits. In contrast, Advantage_system1.1 is found to be affected by additional noise sources for low annealing times, with underlying intrinsic flux noise amplitudes 2 − 3 times higher than in DW_2000Q_6 for all annealing times.