Tankyrase enzymes (TNKS), a core part of the canonical Wnt pathway, are a promising target in the search for potential anti-cancer agents. Although several hundreds of the TNKS inhibitors are currently known, identification of their novel chemotypes attracts considerable interest. In this study, the molecular docking and machine learning-based virtual screening techniques combined with the physico-chemical and ADMET (absorption, distribution, metabolism, excretion, toxicity) profile prediction and molecular dynamics simulations were applied to a subset of the ZINC database containing about 1.7 M commercially available compounds. Out of seven candidate compounds biologically evaluated in vitro for their inhibition of the TNKS2 enzyme using immunochemical assay, two compounds have shown a decent level of inhibitory activity with the IC50 values of less than 10 nM and 10 μM. Relatively simple scores based on molecular docking or MM-PBSA (molecular mechanics, Poisson-Boltzmann, surface area) methods proved unsuitable for predicting the effect of structural modification or for accurate ranking of the compounds based on their binding energies. On the other hand, the molecular dynamics simulations and Free Energy Perturbation (FEP) calculations allowed us to further decipher the structure-activity relationships and retrospectively analyze the docking-based virtual screening performance. This approach can be applied at the subsequent lead optimization stages.