Alongside the evolution of density functional theory into a new era led by the dispersion-corrected hybrid density functional theory approaches, formulation of a new generation of intermolecular potentials has also taken the center stage. An ideal potential formulation should desirably possess simplicity of functional forms, physically meaningful parameters, separability of various terms into atomic-level contributions, computational tractability, ability to capture non-additivity of interactions, transferability across different chemical species, and crucially, chemical fidelity in terms of reproducing the benchmark data. The Lennard-Jones potential, one of the popular intermolecular pair potentials for performing large-scale simulations fails to capture the intricate features of molecular interactions. Woven around the central theme of anisotropy in the nature of intermolecular interactions, herein, we describe various landmark contributions in the quest for chemical fidelity of empirical potential formulations that include (i) incorporation of the anisotropic nature of exchange-repulsion and dispersion contributions, (ii) multipolar description of the dispersion terms, (iii) damping functions to provide an accurate description of the asymptotes, and (iv) transferability of intermolecular interaction terms. We illustrate the nuances of intermolecular force field development in the context of modeling the non-covalent interactions governing the (i) binding of atoms and molecules with carbon nanostructures, (ii) molecular aggregates of polycyclic aromatic hydrocarbons, and (iii) interlayer interactions in layered nanostructures. We exemplify the hierarchy of empirical potentials by depicting them on the various rungs of the Jacob's ladder equivalent of density functional theory for the intermolecular force fields. Finally, we discuss some possible futuristic directions in intermolecular force field development.