Accurate macromolecular structure refinement is of paramount importance in structure based drug discovery as it provides a gateway to using ligand binding free energy calculations and ligand docking techniques. When dealing with high-resolution data, a simple restraint model may be preferred when the data is able to guide atom parameters to an unambiguous location. However, at lower resolution, the additional information contained in a complex force field may aid in refinement by avoiding implausible structures permitted by the simpler restraints. With the advent of the resolution revolution in cryo-electron microscopy, low resolution refinement is common, and likewise increases the need for a reliable force field. Here we report on the incorporation of the OPLS3e force field with the VSGB2.1 solvation model in the popular refinement package Phenix. The implementation is versatile and can be used in both reciprocal and real space refinement, alleviating the need for manually creating accurate ligand restraint dictionaries in the form of CIF files. Our results show significantly improved structure quality at lower resolution for X-ray refinement with reduced ligand strain, while showing only a slight increase in Rfree. For real space refinement of cryo-EM based structures, we find comparable quality structures, goodness-of-fit and reduced ligand strain. In addition, we explicitly show how structure quality is related with the map-model cross correlation as a function of data weight, and how it can be an insightful tool for detecting both over- and underfitting, especially coupled with ligand energies. Further, we have compiled a user-friendly start-to-end script for refining structures with Phenix/OPLS3e, which is available starting in the Schrodinger 2020-3 distribution.