It has been shown that CP violating electron-nucleon and nucleon-nucleon interactions can induce atomic electric dipole moments and are therefore constrained from experimental data. We show that using the experimental upper bounds on neutron and electron electric dipole moments, one can also obtain constraints, in some cases better ones, on these interactions. In addition stringent constraints can also be obtained for muon-quark and tauon-quark four-fermion CP violating interactions, which cannot be constrained from atomic electric dipole moment experiments.