The detection of dark matter is a challenging problem in modern physics. We provide a proposal to detect the coupling of ultralight scalar dark matter to quarks and gluons as well as the coupling of ultralight axion dark matter to gluons with long-baseline atom interferometers. The ultralight scalar and axion dark matter could induce the oscillation of the nuclear charge radii and then oscillate the atomic transition frequency by interacting with quarks and gluons. We calculate the differential phase shift caused by the scalar and axion dark matter in long-baseline atom interferometers and give the proposed constraints on the scalar dark matter coupling parameters $d_g$ and $d_{\hat{m}}$ as well as the axion dark matter coupling parameter $1/f_a$. Our results are expected to improve existing bounds and complement bounds from other experiments in the future.