Over the last few years, many new machine learning-based scoring functions for predicting the binding of small molecules to proteins have been developed. Their objective is to approximate the distribution which takes two molecules as input and outputs the energy of their interaction. This distribution is dependent on the interatomic interactions involved in binding, and only a scoring function that accounts for these interactions can accurately predict binding affinity on unseen molecules. To try to create a method capable of learning these interactions, we built PointVS: a machine learning-based scoring function which achieves state of the art performance even after rigorous filtering of the training set. This filtering is key, as we found that a commonly used benchmark, CASF-16, overestimates the true accuracy of machine learning-based scoring functions when trained using the most commonly used training set. Ranking algorithms using this benchmark rewards memorisation of training data rather than knowledge of the rules of intermolecular binding. We demonstrate that PointVS is able to identify important interactions using attribution, and further, that it can be used to extract important binding pharmacophores from a given protein target when supplied with a number of bound structures. We use this information to perform fragment elaboration, and see improvements in docking scores compared to using structural information from a traditional data-based approach. This not only provides definitive proof that PointVS is learning to identify important binding interactions, but also constitutes the first deep learning-based method for extracting structural information from a target for molecule design.