AGBoost: Attention-based Modification of Gradient Boosting Machine

Konstantinov, Alexander A.; Utkin, Lev V.; R., Kirpichenko, Stanislav

doi:10.23919/fruct54823.2022.9770928

Cited by 7 publications

(12 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Nadaraya-Watson regression in machine learning. There are several machine learning approaches based on applying the Nadaraya-Watson regression [78][79][80][81][82][83][84]. Properties of the boosting with kernel regression estimates as weak learners were studied in [85].…”

Section: Related Workmentioning

confidence: 99%

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya–Watson Regression

Konstantinov

Utkin

2023

Algorithms

Self Cite

View full text Add to dashboard Cite

A new method for estimating the conditional average treatment effect is proposed in this paper. It is called TNW-CATE (the Trainable Nadaraya–Watson regression for CATE) and based on the assumption that the number of controls is rather large and the number of treatments is small. TNW-CATE uses the Nadaraya–Watson regression for predicting outcomes of patients from control and treatment groups. The main idea behind TNW-CATE is to train kernels of the Nadaraya–Watson regression by using a weight sharing neural network of a specific form. The network is trained on controls, and it replaces standard kernels with a set of neural subnetworks with shared parameters such that every subnetwork implements the trainable kernel, but the whole network implements the Nadaraya–Watson estimator. The network memorizes how the feature vectors are located in the feature space. The proposed approach is similar to transfer learning when domains of source and target data are similar, but the tasks are different. Various numerical simulation experiments illustrate TNW-CATE and compare it with the well-known T-learner, S-learner, and X-learner for several types of control and treatment outcome functions. The code of proposed algorithms implementing TNW-CATE is publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya–Watson Regression

Konstantinov

Utkin

2023

Algorithms

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order to extend a set of attention models, several random forest models in which attention mechanisms were incorporated were proposed in [23,34,35]. A gradient boosting machine to which an attention mechanism was added was presented in [36].…”

Section: Related Workmentioning

confidence: 99%

Improved Anomaly Detection by Using the Attention-Based Isolation Forest

et al. 2022

Self Cite

View full text Add to dashboard Cite

A new modification of the isolation forest called the attention-based isolation forest (ABIForest) is proposed for solving the anomaly detection problem. It incorporates an attention mechanism in the form of Nadaraya–Watson regression into the isolation forest to improve the solution of the anomaly detection problem. The main idea underlying the modification is the assignment of attention weights to each path of trees with learnable parameters depending on the instances and trees themselves. Huber’s contamination model is proposed to be used to define the attention weights and their parameters. As a result, the attention weights are linearly dependent on learnable attention parameters that are trained by solving a standard linear or quadratic optimization problem. ABIForest can be viewed as the first modification of the isolation forest to incorporate an attention mechanism in a simple way without applying gradient-based algorithms. Numerical experiments with synthetic and real datasets illustrate that the results of ABIForest outperform those of other methods. The code of the proposed algorithms has been made available.

show abstract

“…The Nadaraya-Watson regression in machine learning. There are several machine learning approaches based on applying the Nadaraya-Watson regression [74,75,76,77,78,79,80]. Properties of the boosting with kernel regression estimates as weak learners were studied in [81].…”

Section: Related Workmentioning

confidence: 99%

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya-Watson Regression

Konstantinov¹,

R.²,

Utkin³

2022

Preprint

View full text Add to dashboard Cite

A new method for estimating the conditional average treatment effect is proposed in the paper. It is called TNW-CATE (the Trainable Nadaraya-Watson regression for CATE) and based on the assumption that the number of controls is rather large whereas the number of treatments is small. TNW-CATE uses the Nadaraya-Watson regression for predicting outcomes of patients from the control and treatment groups. The main idea behind TNW-CATE is to train kernels of the Nadaraya-Watson regression by using a weight sharing neural network of a specific form. The network is trained on controls, and it replaces standard kernels with a set of neural subnetworks with shared parameters such that every subnetwork implements the trainable kernel, but the whole network implements the Nadaraya-Watson estimator. The network memorizes how the feature vectors are located in the feature space. The proposed approach is similar to the transfer learning when domains of source and target data are similar, but tasks are different. Various numerical simulation experiments illustrate TNW-CATE and compare it with the well-known T-learner, S-learner and X-learner for several types of the control and treatment outcome functions. The code of proposed algorithms implementing TNW-CATE is available in https://github.com/Stasychbr/TNW-CATE.

show abstract

AGBoost: Attention-based Modification of Gradient Boosting Machine

Cited by 7 publications

References 27 publications

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya–Watson Regression

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya–Watson Regression

Improved Anomaly Detection by Using the Attention-Based Isolation Forest

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya-Watson Regression

Contact Info

Product

Resources

About