Estrogen receptor α (ERα) plays a significant role in occurrence of breast cancer and may cause various adverse side-effects when ERα is an off-target protein. A theoretical model was derived to predict the binding affinity of ERα using the pharmacophore ensemble/support vector machine (PhE/SVM) scheme to consider the promiscuous characteristic of ERα. The estimations by PhE/SVM were discovered to be in good agreement with the observed values for those training molecules ( n = 31, r = 0.80, q = 0.77, RMSE = 0.57, s = 0.58), test molecules ( n = 179, q = 0.91-0.96, RMSE = 0.33, s = 0.26) and outliers ( n = 15, q = 0.80-0.86, RMSE = 0.56, s = 0.49). When subjected to various statistical validations, the PhE/SVM model consistently fulfilled the strictest criteria. A mock test also asserted its predictivity. When compared with crystal structures, the calculated results are consistent with the reported ERα-ligand co-complex structure, and the plasticity nature of ERα is also disclosed. Consequently, this precise, fast, and robust model can be adopted to predict ERα-ligand binding affinities and to design safer non-ERα-targeted pharmaceuticals in the process of drug discovery and development.