deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

Ulmer, Dennis; Hardmeier, Christian; Frellsen, Jes

doi:10.48550/arxiv.2204.06815

Cited by 5 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To demonstrate how significant the superiority of the model trained on a signal from the central and front EEG channels (C3-M2, CZ-O1, F3-M2, and F4-M1) was over the back channels (O1-M2 and O2-M1), we employed the almost stochastic order (ASO) significance test [ 37 , 38 ], as implemented by [ 39 ]. The method compares scores from two deep learning models

and

by computing the significance score

.…”

Section: Discussionmentioning

confidence: 99%

Influence of Channel Selection and Subject’s Age on the Performance of the Single Channel EEG-Based Automatic Sleep Staging Algorithms

Nazih

Shahin

Eldesouki

et al. 2023

Sensors

View full text Add to dashboard Cite

The electroencephalogram (EEG) signal is a key parameter used to identify the different sleep stages present in an overnight sleep recording. Sleep staging is crucial in the diagnosis of several sleep disorders; however, the manual annotation of the EEG signal is a costly and time-consuming process. Automatic sleep staging algorithms offer a practical and cost-effective alternative to manual sleep staging. However, due to the limited availability of EEG sleep datasets, the reliability of existing sleep staging algorithms is questionable. Furthermore, most reported experimental results have been obtained using adult EEG signals; the effectiveness of these algorithms using pediatric EEGs is unknown. In this paper, we conduct an intensive study of two state-of-the-art single-channel EEG-based sleep staging algorithms, namely DeepSleepNet and AttnSleep, using a recently released large-scale sleep dataset collected from 3984 patients, most of whom are children. The paper studies how the performance of these sleep staging algorithms varies when applied on different EEG channels and across different age groups. Furthermore, all results were analyzed within individual sleep stages to understand how each stage is affected by the choice of EEG channel and the participants’ age. The study concluded that the selection of the channel is crucial for the accuracy of the single-channel EEG-based automatic sleep staging methods. For instance, channels O1-M2 and O2-M1 performed consistently worse than other channels for both algorithms and through all age groups. The study also revealed the challenges in the automatic sleep staging of newborns and infants (1–52 weeks).

show abstract

and

by computing the significance score

.…”

Section: Discussionmentioning

confidence: 99%

Influence of Channel Selection and Subject’s Age on the Performance of the Single Channel EEG-Based Automatic Sleep Staging Algorithms

Nazih

Shahin

Eldesouki

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…To perform significance tests we used the Almost Stochastic Order (ASO) test [36,37] with a 95% confidence level (α = 0.05). Each ASO test outputs a violation error min which denotes the degree to which the hypothesis that "method A is always better than method B" is being violated.…”

Section: Significance Testsmentioning

confidence: 99%

On Retrospective k-space Subsampling schemes For Deep MRI Reconstruction

Yiasemis¹,

Sánchez²,

Sonke³

et al. 2023

Preprint

View full text Add to dashboard Cite

Purpose: The MRI k-space acquisition is time consuming. Traditional techniques aim to acquire accelerated data, which in conjunction with recent DL methods, aid in producing high-fidelity images in truncated times. Conventionally, subsampling the k-space is performed by utilizing Cartesianrectilinear trajectories, which even with the use of DL, provide imprecise reconstructions, though, a plethora of non-rectilinear or non-Cartesian trajectories can be implemented in modern MRI scanners. This work investigates the effect of the k-space subsampling scheme on the quality of reconstructed accelerated MRI measurements produced by trained DL models. Methods:The RecurrentVarNet was used as the DL-based MRI-reconstruction architecture. Cartesian fully-sampled multi-coil k-space measurements from three datasets with different accelerations were retrospectively subsampled using eight distinct subsampling schemes (four Cartesian-rectilinear, two Cartesian non-rectilinear, two non-Cartesian). Experiments were conducted in two frameworks: Scheme-specific, where a distinct model was trained and evaluated for each dataset-subsampling scheme pair, and multi-scheme, where for each dataset a single model was trained on data randomly subsampled by any of the eight schemes and evaluated on data subsampled by all schemes.Results: In the scheme-specific setting RecurrentVarNets trained and evaluated on non-rectilinearly subsampled data demonstrated superior performance especially for high accelerations, whilst in the multi-scheme setting, reconstruction performance on rectilinearly subsampled data improved when compared to the scheme-specific experiments. Conclusion:Training DL-based MRI reconstruction algorithms on non-rectilinearly subsampled measurements can produce more faithful reconstructions. Our findings demonstrate the potential for using DL-based methods trained on prospective acquisitions with non-rectilinearly subsampled measurements to optimize scan time and image quality.

show abstract

“…Model comparisons are performed using the almost stochastic order test. Following the recommendation of Ulmer et al (2022), τ = 0.2 is used as a threshold for the significance of the superiority of one model over another. Moreover, the error level α = 0.05 is already included in the min calculation.…”

Section: Parameters Of Statistical Testsmentioning

confidence: 99%

“…The smaller min , the more certain is the superiority of model A. Here, τ = 0.2 is recommended as an acceptance limit to achieve a balance between false positives and false negatives (Ulmer et al (2022)), although the min result should always be included for the sake of transparency. Note, however, that min is not a p value, unlike other statistical tests, but the significance level α is already included in the calculation of min .…”

Section: Structurementioning

confidence: 99%

evalPM: a framework for evaluating machine learning models for particulate matter prediction

Woltmann,

Deepe,

Hartmann

et al. 2023

Environ Monit Assess

View full text Add to dashboard Cite

Air pollution through particulate matter (PM) is one of the largest threats to human health. To understand the causes of PM pollution and enact suitable countermeasures, reliable predictions of future PM concentrations are required. In the scientific literature, many methods exist for machine learning (ML)-based PM prediction, though their quality is difficult to compare because, among other things, they use different data sets and evaluate the resulting predictions differently. For a new data set, it is not apparent which of the existing prediction methods is best suited. In order to ease the assessment of said models, we present evalPM, a framework to easily create, evaluate, and compare different ML models for immission-based PM prediction. To achieve this, the framework provides flexibility regarding data sets, input features, target variables, model types, hyperparameters, and model evaluation. It has a modular design consisting of several components, each providing at least one required flexibility. The individual capabilities of the framework are demonstrated using 16 different models from the related literature by means of temporal prediction of PM concentrations for four European data sets, showing the capabilities and advantages of the evalPM framework. In doing so, it is shown that the framework allows fast creation and evaluation of ML-based PM prediction models.

show abstract

deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

Cited by 5 publications

References 19 publications

Influence of Channel Selection and Subject’s Age on the Performance of the Single Channel EEG-Based Automatic Sleep Staging Algorithms

Influence of Channel Selection and Subject’s Age on the Performance of the Single Channel EEG-Based Automatic Sleep Staging Algorithms

On Retrospective k-space Subsampling schemes For Deep MRI Reconstruction

evalPM: a framework for evaluating machine learning models for particulate matter prediction

Contact Info

Product

Resources

About