We study numerically the pair trajectories of rigid circular particles in a two dimensional inertialess simple shear flow of a (Binghamian) yield stress fluid. We use a Lagrange multiplier based fictitious domain method, following Glowinski et al. [26, 28, 29], for solving the problem. Contacts between the particles at a finite interparticle distance interpreted as a roughness are taken into account with the da Cunha and Hinch [12] model. Another model, introduced by Glowinski et al. [27], is shown to provide similar results in the limit of infinite contact stiffness. Due to the nonlinear behavior of the suspending fluid, it is found that the trajectories of the particles depend on the shear rate, the relevant dimensionless parameter being here the Bingham number, which compares plastic forces to viscous forces. In absence of interparticle contacts, fore-aft symmetry is observed in all cases; however, the particles are found to come closer to each other as the Bingham number is increased: the plastic behavior of the suspending fluid decreases the range of hydrodynamic interactions. As contacts are introduced, fore-aft asymmetry is observed. Plastic effects are found to enhance surface roughness effects: contacts between particles occur for smaller surface roughness at large Bn. Moreover, the magnitude of the asymmetry is increased as the Bingham number is increased. These observations may explain why the microstructure of suspensions of particles in a yield stress fluid is shear-rate-dependent [45] leading to a complex nonlinear macroscopic behavior.