Accurate prediction of binding affinities from protein-ligand atomic coordinates remains a major challenge in early stages of drug discovery. Using modular message passing graph neural networks describing both the ligand and the protein in their free and bound states, we unambiguously evidence that explicit description of protein-ligand non-covalent interactions does not provide any advantage with respect to ligand or protein descriptors. Simple models, inferring binding affinities of test samples from that of the closest ligands or proteins in the training set, already exhibit good performances suggesting that memorization largely dominates true learning in the deep neural networks. The current study suggests considering only non-covalent interactions while omitting their protein and ligand atomic environments. Removing all hidden biases probably require much denser protein-ligand training matrices and a coordinated effort of the drug design community to solve the necessary protein-ligand structures.

Generative models are being increasingly used in drug discovery campaigns, very often coupled with ADME or bio-assays Quantitative Structure-Activity Relationship (QSAR) models to optimize a given set of properties. The molecules proposed by these algorithms are often revealed to be false positives, i.e. outside the true candidate drug target profile (CDTP), because the predictors are being hacked by the generative model during the optimization. By hacking we mean an over-optimization of the predicted score leading to an actual decrease or stagnation of the real score. This issue is reminiscent of adversarial examples in Machine Learning and it can be seen as evidence of the Goodhart’s law - “when a measure becomes a target, it ceases to be a good measure”. This issue is even more apparent in a multi-objective setting where the models need to extrapolate outside the train set distribution, because there are no known molecules satisfying all the objectives simultaneously in the initial train set. However, analysing this problem is very difficult and expensive since it requires synthesis and tests of the generated molecules. Consequently, efforts have been made to develop various kinds of in silico oracles - real-valued functions used as proxies for molecular properties, to help with the evaluation of these generative model-based pipelines. However, these oracles have had a limited value so far, as they are often too easy to model in comparison to biological assays, and are usually limited to mono-objective cases. In this work, we introduce a simulator of multi-target assays using a smartly initialized neural network (NN) which returns continuous values for any input molecule. We use this oracle to replicate a real-world prospective lead optimization scenario. First, we train predictive models on an initial small sample of molecules aimed at predicting their real oracle values. Second, we generate new optimized molecules using the open-source GuacaMol package, coupled with the previously built models. Finally, we select compounds which match the CDTP according to the predicted values and evaluate them by computing the true oracle values. We observe that even when the predictive models have excellent estimated performance metrics, the final selection still contains many false positives according to the NN-based oracle. We evaluate the optimization behavior in mono and bi-objective scenarios using either a logistic regression or a random forest predictive model. We also propose and evaluate several methods to help mitigate the hacking issue.