Interacting particle systems play a key role in science and engineering. Access to the governing particle interaction law is fundamental for a complete understanding of such systems. However, the inherent system complexity keeps the particle interaction hidden in many cases. Machine learning methods have the potential to learn the behavior of interacting particle systems by combining experiments with data analysis methods. However, most existing algorithms focus on learning the kinetics at the particle level. Learning pairwise interaction, e.g., pairwise force or pairwise potential energy, remains an open challenge. Here, we propose an algorithm that adapts the Graph Networks framework, which contains an edge part to learn the pairwise interaction and a node part to model the dynamics at particle level. Different from existing approaches that use neural networks in both parts, we design a deterministic operator in the node part. The designed physics operator on the nodes restricts the output space of the edge neural network to be exactly the pairwise interaction. We test the proposed methodology on multiple datasets and demonstrate that it achieves considerably better performance in inferring correctly the pairwise interactions while also being consistent with the underlying physics on all the datasets than existing purely data-driven models. The developed methodology can support a better understanding and discovery of the underlying particle interaction laws, and hence guide the design of materials with targeted properties.Recent efforts on developing machine learning (ML) methods for the discovery of particle interaction laws have shown great potential in overcoming these challenges [5][6][7][8][9][10][11]. These ML methods, such as the Graph Network-based Simulators (GNS) [12] for simulating physical processes, Dynamics Extraction From cryo-em Map (DEFMap) [13] for learning the atomic fluctuation in proteins, the SchNet [14,15] which can learn the molecular energy and the neural relational inference model (NRI) [16] developed for inferring heterogeneous interactions, can be applied on various types of interacting particle systems such as water particles, sand and plastically deformable particles. They allow implicit and explicit learning of the mechanical behavior of particle systems without prior Preprint. Under review.