Evaluation Methods and Measures for Causal Learning Algorithms

Cheng, Lu; Guo, Ruocheng; Moraffah, Raha; Sheth, Paras; Candan, K. Selçuk; Huan, Liu

doi:10.1109/tai.2022.3150264

Cited by 39 publications

(15 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Currently, although there are a number of software packages for treatment effect estimation using observational data, e.g., Python library CausalML, EconML, DoWhy, software for validating the (C)ATEs obtained from observational dataset by comparison to RCT data are, to our best knowledge, non-existent. Earlier work by Wendling et al (2018), Alaa and Van Der Schaar (2019), Schuler et al (2017), Powers et al (2018), Franklin et al (2014), and Cheng et al (2022), and existing software packages such as the R package MethodEvaluation (Schuemie et al 2020), the Python package Causality-Benchmark (Shimoni et al 2018), and the Python package JustCause (Franz 2020), do approximate a data generation process for a given observational dataset, and use simulation methods for treatment effect validation. An overview of existing software for treatment effect estimation and validation is provided in table 1.…”

Section: Related Workmentioning

confidence: 99%

RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects

Shen

Geleijnse

Kaptein

2023

Preprint

View full text Add to dashboard Cite

Despite the recent development of numerous methods aiming to estimate individual-level treatment effects based on observational data, understanding the validity of these estimates remains challenging. Often it is unclear whether the observational data meet the assumptions imposed by the method. Additionally, there is often large flexibility in model choice when implementing a given method. We present the R package RCTrep designed to make it easy to compare and validate estimates of (conditional) average treatment effects obtained using observational data by a) making it easy to obtain and visualize estimates derived using a large variety of methods, and b) ensuring that estimates are easily compared to a gold standard (i.e., estimates derived from randomized controlled trials). RCTrep offers a generic protocol for treatment effect validation based on four simple steps, namely, set-selection, estimation, diagnosis, and validation. The package provides a simple dashboard to review the obtained results. This article presents design of RCTrep and serves as a user guide for researchers aiming to leverage the potential of observational data.

show abstract

Section: Related Workmentioning

confidence: 99%

RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects

Shen

Geleijnse

Kaptein

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…One of the biggest open problems in CausalML is the lack of public benchmark resources to train and evaluate causal models. Cheng et al [348] found that the reason for this lack of benchmarks is the difficulty of observing interventions in the real world, because the necessary experimental conditions in form of randomized con-trol trials (RCTs) are often expensive, unethical or time-consuming. In other words, collecting interventional data involves actively interacting with an environment (i.e.…”

Section: Lack Of Benchmarksmentioning

confidence: 99%

Causal Machine Learning: A Survey and Open Problems

Jean¹,

Lynch²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Causal Machine Learning (CausalML) is an umbrella term for machine learning methods that formalize the data-generation process as a structural causal model (SCM). This allows one to reason about the effects of changes to this process (i.e., interventions) and what would have happened in hindsight (i.e., counterfactuals). We categorize work in CausalML into five groups according to the problems they tackle: (1) causal supervised learning, (2) causal generative modeling, (3) causal explanations, (4) causal fairness, (5) causal reinforcement learning. For each category, we systematically compare its methods and point out open problems. Further, we review modality-specific applications in computer vision, natural language processing, and graph representation learning. Finally, we provide an overview of causal benchmarks and a critical discussion of the state of this nascent field, including recommendations for future work.

show abstract

“…Causal inference plays a significant role in improving the correctness and interpretability of deep learning systems [ 46 , 47 , 48 ]. Researchers focus on using causal inference in the reinforcement learning community to mitigate estimation errors of off-policy evaluation in partially observable environments [ 49 , 50 , 51 ].…”

Section: Related Work: Smart Grids Communications In Cooperative Agen...mentioning

confidence: 99%

Pruning the Communication Bandwidth between Reinforcement Learning Agents through Causal Inference: An Innovative Approach to Designing a Smart Grid Power System

Zhang

Liu

Li³

et al. 2022

Sensors

View full text Add to dashboard Cite

Electricity demands are increasing significantly and the traditional power grid system is facing huge challenges. As the desired next-generation power grid system, smart grid can provide secure and reliable power generation, and consumption, and can also realize the system’s coordinated and intelligent power distribution. Coordinating grid power distribution usually requires mutual communication between power distributors to accomplish coordination. However, the power network is complex, the network nodes are far apart, and the communication bandwidth is often expensive. Therefore, how to reduce the communication bandwidth in the cooperative power distribution process task is crucially important. One way to tackle this problem is to build mechanisms to selectively send out communications, which allow distributors to send information at certain moments and key states. The distributors in the power grid are modeled as reinforcement learning agents, and the communication bandwidth in the power grid can be reduced by optimizing the communication frequency between agents. Therefore, in this paper, we propose a model for deciding whether to communicate based on the causal inference method, Causal Inference Communication Model (CICM). CICM regards whether to communicate as a binary intervention variable, and determines which intervention is more effective by estimating the individual treatment effect (ITE). It offers the optimal communication strategy about whether to send information while ensuring task completion. This method effectively reduces the communication frequency between grid distributors, and at the same time maximizes the power distribution effect. In addition, we test the method in StarCraft II and 3D environment habitation experiments, which fully proves the effectiveness of the method.

show abstract

Evaluation Methods and Measures for Causal Learning Algorithms

Cited by 39 publications

References 72 publications

RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects

RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects

Causal Machine Learning: A Survey and Open Problems

Pruning the Communication Bandwidth between Reinforcement Learning Agents through Causal Inference: An Innovative Approach to Designing a Smart Grid Power System

Contact Info

Product

Resources

About