Motivation: Reverse engineering of gene regulatory networks has for years struggled with high correlation in expression between regulatory elements. If two regulators have matching expression patterns it is impossible to differentiate between the two, and thus false positive identifications are abundant. Results: To allow for gene regulation predictions of high confidence, we propose a novel method, LiPLike, that assumes a regression model and iteratively searches for interactions that cannot be replaced by a linear combination of other predictors. To compare the performance of LiPLike with other available inference methods, we benchmarked LiPLike using three independent datasets from the previous DREAM5 challenge. We found that LiPLike could be used to stratify predictions of other inference tools, and when applied to the predictions of DREAM5 participants we observed the accuracy to on average be improved >140% compared to individual methods. Furthermore, we observed that LiPLike independently predicted networks better than all DREAM5 participants when applied to biological data. When predicting the Escherichia coli network, LiPLike had an accuracy of 0.38 for the top-ranked 100 interactions, whereas the corresponding DREAM5 consensus model yielded an accuracy of 0.11. Availability: We made LiPLike available to the community as a Python toolbox, available at https://gitlab.com/Gustafsson-lab/liplike. We believe that LiPLike will be used for high confidence predictions in studies where individual model interactions are of high importance, and that LiPLike will be used to remove false positive predictions made by other state-of-the-art gene-gene regulation prediction tools.