The accurate prediction of binding free energy is a major challenge in structure-based drug design. Quantum mechanics (QM)-based approaches show promising potential in predicting ligand–protein binding affinity by accurately describing the behavior and structure of electrons. However, traditional QM calculations face computational limitations, hindering their practical application in drug design. Nevertheless, the fragment molecular orbital (FMO) method has gained widespread application in drug design due to its ability to reduce computational costs and achieve efficient ab initio QM calculations. Although the FMO method has demonstrated its reliability in calculating the gas phase potential energy, the binding of proteins and ligands also involves other contributing energy terms, such as solvent effects, the ‘deformation energy’ of a ligand’s bioactive conformations, and entropy. Particularly in cases involving ionized fragments, the calculation of solvation free energy becomes particularly crucial. We conducted an evaluation of some previously reported implicit solvent methods on the same data set to assess their potential for improving the performance of the FMO method. Herein, we develop a new QM-based binding free energy calculation method called FMOScore, which enhances the performance of the FMO method. The FMOScore method incorporates linear fitting of various terms, including gas-phase potential energy, deformation energy, and solvation free energy. Compared to other widely used traditional prediction methods such as FEP+, MM/PBSA, MM/GBSA, and Autodock vina, FMOScore showed good performance in prediction accuracies. By constructing a retrospective case study, it was observed that incorporating calculations for solvation free energy and deformation energy can further enhance the precision of FMO predictions for binding affinity. Furthermore, using FMOScore-guided lead optimization against Src homology-2-containing protein tyrosine phosphatase 2 (SHP-2), we discovered a novel and potent allosteric SHP-2 inhibitor (compound 8).