Developing deterministic neighborhood-informed point-particle closure models using machine learning has garnered interest in recent times from dispersed multiphase flow community. The robustness of neural models for this complex multi-body problem is hindered by the availability of particle-resolved data. The present work addresses this unavoidable limitation of data paucity by implementing two strategies: (i) by using a rotation and reflection equivariant neural network and (ii) by pursuing a physics-based hierarchical machine learning approach. The resulting machine learned models are observed to achieve a maximum accuracy of 85% and 96% in the prediction of neighborinduced force and torque fluctuations, respectively, for a wide range of Reynolds number and volume fraction conditions considered. Furthermore, we pursue force and torque network architectures that provide universal prediction spanning a wide range of Reynolds number (0.25 ≤ Re ≤ 250) and particle volume fraction (0 ≤ φ ≤ 0.4). The hierarchical nature of the approach enables improved prediction of quantities such as streamwise torque, by going beyond binary interactions to include trinary interactions.