The first- and second-generation clinically used HIV-1 integrase (IN) strand transfer inhibitors (INSTIs) are key components of antiretroviral therapy (ART), which work by blocking the integration step in the HIV-1 replication cycle that is catalyzed by a nucleoprotein assembly called an intasome. However, resistance to even the latest clinically used INSTIs is beginning to emerge. Developmental third-generation INSTIs, based on naphthyridine scaffolds, are promising candidates to combat drug-resistant viral variants. Among these novel INSTIs, compound 4f exhibits two distinct conformations when binding with intasomes from HIV-1 and the closely related prototype foamy virus (PFV) despite the high structural similarity of their INSTI binding pockets. The molecular mechanism and the key active site residues responsible for these differing binding modes in closely related intasomes remain elusive. To unravel the molecular determinants governing the two distinct binding modes, we applied a novel molecular dynamics-based free energy method that utilizes alchemical pathways to overcome the sampling challenges associated with transitioning between the two bound conformations of ligand 4f within the crowded environments of the INSTI binding pockets in these intasomes. The calculated conformational free energies successfully recapitulate the experimentally observed binding mode preferences in the two viral intasomes. Analysis of the simulated structures suggests that the observed binding mode preferences are caused by amino acid residue differences in both the front and the central catalytic sub-pocket of the INSTI binding site in HIV-1 and PFV. Additional free energy calculations on mutants of HIV-1 and PFV revealed that while both sub-pockets contribute to binding mode selection, the central sub-pocket plays a more important role. These results highlight the importance of both side chain and solvent reorganization, as well as the conformational entropy in determining the ligand binding mode, and will help inform the development of more effective INSTIs for combatting drug-resistant viral variants.