REPR: Rule-Enhanced Penalized Regression

Abstract

This article describes a new rule-enhanced penalized regression procedure for the generalized regression problem of predicting scalar responses from observation vectors in the absence of a preferred functional form. It enhances standard L 1-penalized regression by adding dynamically generated rules, that is, new 0-1 covariates, corresponding to multidimensional "box" sets. In contrast to prior approaches to this class of problems, we draw heavily on standard (but non-polynomial-time) mathematical programming techniques, enhanced by parallel computing. We identify and incorporate new rules using a form of classical column generation and solve the resulting pricing subproblem, which is NP-hard, either exactly by a specialized parallel branch-and-bound method or by a greedy heuristic based on Kadane's algorithm. The resulting rule-enhanced regression method can be computation intensive when we solve the subproblems exactly, but our computational tests suggest that it outperforms prior methods at making accurate and stable predictions from relatively small data samples. Through selective use of our greedy heuristic, we can make our method's run time generally competitive with some established methods, without sacrificing prediction performance. We call our method's pricing subproblem rectangular maximum agreement. History: This paper has been accepted for the INFORMS Journal on Optimization Special Issue on Machine Learning and Optimization.

Rule Induction in Knowledge Graphs Using Linear Programming

“…We propose an LP model inspired by LP boosting methods for classification using classical column generation techniques (Demiriz, Bennett, and Shawe-Taylor 2002;Eckstein and Goldberg 2012;Eckstein, Kagawa, and Goldberg 2019;Dash, Günlük, and Wei 2018). Our goal is to create a weighted linear combination of first-order logic rules to be used as a scoring function for KGC.…”

Rule Induction in Knowledge Graphs Using Linear Programming

“…We propose a linear programming model which is inspired by LP boosting methods for classification using classical column generation techniques [Demiriz et al, 2002, Golderg, 2012, Eckstein and Goldberg, 2012, Eckstein et al, 2019, Dash et al, 2018. The goal of our model is to create a weighted combination of first-order logic rules (similar to the one proposed in Yang et al [2017]) to be used as a prediction function for the task of knowledge graph link prediction.…”

Multilinear Sets with Two Monomials and Cardinality Constraints

“…Note that for ( 14), π j + δ0 ≤ 1 for each j ∈ J. Therefore, j∈Q π j + n δ0 ≤ j∈J (π j + δ0 ) ≤ n, i.e., (14) can never be violated if u = n and (3)-( 13) are satisfied. Similar arguments can be made for each of ( 14)-( 18) when u = n and for each of ( 19)-( 23) when l = 0.…”

“…where L, U are nonnegative integers. The problem of minimizing a linear function over X contains as a special case the maximum monomial agreement problem, and a cardinality constrained version of it, which has been analyzed in the context of machine learning [11,13,14,12], and solved via branch-and-bound methods and heuristics. An explicit polyhedral description of the convex hull when the sets S i are nested was given in [3].…”