Lorentz symmetry is an important concept in modern physics. Precision pulsar timing was used to put tight constraints on the coefficients for Lorentz violation in the pure-gravity sector of the Standard-Model Extension (SME). We extend the analysis to Lorentz-violating matter-gravity couplings, utilizing three small-eccentricity relativistic neutron star (NS) -white dwarf (WD) binaries. We obtain compelling limits on various SME coefficients related to the neutron, the proton, and the electron. These results are complementary to limits obtained from lunar laser ranging and clock experiments.