Simulations evaluating resampling methods for causal discovery: ensemble performance and calibration

Kummerfeld, Erich; Rix, Alexander

doi:10.1109/bibm47256.2019.8983327

Cited by 11 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bootstrap is often used to assess the variability in an estimation. In the causal discovery literature, the frequency of discovering an edge in bootstrap samples has been shown to be a good indicator for the presence of the edge in the true network [ 42 ]. Therefore, we propose to estimate the network difference by incorporating the confidence of estimating individual networks using bootstrap.…”

Section: Methods For Estimating the Structural Differences Between Pa...mentioning

confidence: 99%

“…The resampling methods are directly related to the goal of the current study, except we are not only interested in bootstrap frequency for the discovered edges (which relates to false discovery rate), but also the pair of nodes for which no edge was identified between them (relates to false negatives). The resampling was first proposed for assessing confidence intervals for causal discovery in [38,39], and was empirically demonstrated to adequate estimate confidence for causal discovery in simulated datasets using various causal discovery algorithms [38][39][40][41][42]. Therefore, we resort to resampling to estimate the confidence of causal discovery and to support the comparison between the inferred networks.…”

Section: Relevant Prior Literaturementioning

confidence: 99%

See 1 more Smart Citation

Comparing Causal Bayesian Networks Estimated from Data

Ma,

Tourani

2024

Entropy

View full text Add to dashboard Cite

The knowledge of the causal mechanisms underlying one single system may not be sufficient to answer certain questions. One can gain additional insights from comparing and contrasting the causal mechanisms underlying multiple systems and uncovering consistent and distinct causal relationships. For example, discovering common molecular mechanisms among different diseases can lead to drug repurposing. The problem of comparing causal mechanisms among multiple systems is non-trivial, since the causal mechanisms are usually unknown and need to be estimated from data. If we estimate the causal mechanisms from data generated from different systems and directly compare them (the naive method), the result can be sub-optimal. This is especially true if the data generated by the different systems differ substantially with respect to their sample sizes. In this case, the quality of the estimated causal mechanisms for the different systems will differ, which can in turn affect the accuracy of the estimated similarities and differences among the systems via the naive method. To mitigate this problem, we introduced the bootstrap estimation and the equal sample size resampling estimation method for estimating the difference between causal networks. Both of these methods use resampling to assess the confidence of the estimation. We compared these methods with the naive method in a set of systematically simulated experimental conditions with a variety of network structures and sample sizes, and using different performance metrics. We also evaluated these methods on various real-world biomedical datasets covering a wide range of data designs.

show abstract

Section: Methods For Estimating the Structural Differences Between Pa...mentioning

confidence: 99%

Section: Relevant Prior Literaturementioning

confidence: 99%

Comparing Causal Bayesian Networks Estimated from Data

Ma,

Tourani

2024

Entropy

View full text Add to dashboard Cite

show abstract

“…Resampling is a statistical approach that uses random extract samples from a dataset to build a new dataset with fewer or more samples that have the same distribution as the original dataset [38], [39], [40], [41]. Resampling is used to generate a more representative sample of the population, improve the performance of a model, or balance an unbalanced dataset.…”

Section: Resamplingmentioning

confidence: 99%

Improving CADx System Performance for Skin Disease Detection using Ensemble Machine Learning Models

Asaduzzaman,

Thompson,

Uddin

2024

Preprint

View full text Add to dashboard Cite

Conventional computer-aided diagnosis (CADx) systems play a crucial role in assisting medical professionals with the detection of skin diseases. However, these systems often involve manual, time-consuming, and error-prone processes. Recent studies show that machine learning models have potential to improve the accuracy of CADx systems. In this work, we present research findings aimed at improving the performance of CADx systems for detecting skin diseases by applying ensemble machine learning models. The investigation encompasses the exploration of three popular classification methods: linear discriminant analysis (LDA), support vector machine (SVM), and convolutional neural network (CNN); and an ensemble model of CNN with SVM. The HAM10000 dataset from Kaggle is used to train and test all classification models. Resampling is employed to address class imbalance in the dataset. Through rigorous experiments, the results highlight the compelling efficacy of the ensemble CNNSVM model, unveiling heightened accuracy up to 92% (from CNN accuracy 85% and SVM accuracy 83%). The outcome of this work has profound implications for artificial intelligence (AI) accelerated medical domains in advancing the accuracy and efficiency of skin disease treatment.

show abstract

“…Previous work on the finite sample performance of causal discovery algorithms has been primarily about comparing different causal discovery algorithms according to their performance on different learning problems rather than assessing questions such as how many samples are required to get adequate performance [6,[38][39][40]. These evaluations have almost exclusively relied on simulation studies, with data generated in silico from either randomly generated causal models or expert-made causal models of real-world processes found in model repositories.…”

Section: Causal Discoverymentioning

confidence: 99%

Power Analysis for Causal Discovery

Kummerfeld

Williams

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Causal discovery algorithms have the potential to impact many fields of science. However, substantial foundational work on the statistical properties of causal discovery algorithms is still needed. This paper presents what is to our knowledge the first method for conducting power analysis for causal discovery algorithms. The power sample characteristics of causal discovery algorithms typically cannot be described by a closed formula, but we resolve this problem by developing a new power sample analysis method based on standardized in silico simulation experiments. Our procedure generates data with carefully controlled statistical effect sizes in order to enable an accurate numerical power sample analysis. We present that method, apply it to generate an initial power analysis table, provide a web interface for searching this table, and show how the table or web interface can be used to solve several types of real world power analysis problems, such as sample size planning, interpretation of results, and sensitivity analysis.

show abstract

Simulations evaluating resampling methods for causal discovery: ensemble performance and calibration

Cited by 11 publications

References 16 publications

Comparing Causal Bayesian Networks Estimated from Data

Comparing Causal Bayesian Networks Estimated from Data

Improving CADx System Performance for Skin Disease Detection using Ensemble Machine Learning Models

Power Analysis for Causal Discovery

Contact Info

Product

Resources

About