Blocked 3×2 Cross-Validated <i>t</i>-Test for Comparing Supervised Classification Learning Algorithms

Wang, Yu; Wang, Ruibo; Huichen, Jia; Li, Jihong

doi:10.1162/neco_a_00532

Cited by 22 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, Student’s t -tests are performed to test the equality (null hypothesis) of the average F-scores between the best and the baseline classifiers obtained in the training step. Student’s t -tests are the most recommended and used statistical tests to compare machine learning models [ 25 ].…”

Section: Analysis Frameworkmentioning

confidence: 99%

Interpretable prediction of brain activity during conversations from multimodal behavioral signals

Hmamouche,

Ochs,

Prévot

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

We present an analytical framework aimed at predicting the local brain activity in uncontrolled experimental conditions based on multimodal recordings of participants’ behavior, and its application to a corpus of participants having conversations with another human or a conversational humanoid robot. The framework consists in extracting high-level features from the raw behavioral recordings and applying a dynamic prediction of binarized fMRI-recorded local brain activity using these behavioral features. The objective is to identify behavioral features required for this prediction, and their relative weights, depending on the brain area under investigation and the experimental condition. In order to validate our framework, we use a corpus of uncontrolled conversations of participants with a human or a robotic agent, focusing on brain regions involved in speech processing, and more generally in social interactions. The framework not only predicts local brain activity significantly better than random, it also quantifies the weights of behavioral features required for this prediction, depending on the brain area under investigation and on the nature of the conversational partner. In the left Superior Temporal Sulcus, perceived speech is the most important behavioral feature for predicting brain activity, regardless of the agent, while several features, which differ between the human and robot interlocutors, contribute to the prediction in regions involved in social cognition, such as the TemporoParietal Junction. This framework therefore allows us to study how multiple behavioral signals from different modalities are integrated in individual brain regions during complex social interactions.

show abstract

Section: Analysis Frameworkmentioning

confidence: 99%

Interpretable prediction of brain activity during conversations from multimodal behavioral signals

Hmamouche,

Ochs,

Prévot

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…In this article, we employ a deep learning model evaluation technique known as “5% cross-validation” ( Yu et al, 2014 ) to assess the performance of the SA-BO-CNN model and determine suitable parameters for classification training. This method involves processing Excel table data and adjusting the input data format to match the SA-BO-CNN model’s requirements.…”

Section: Intrusion Detection System Based On Convolution Neural Networkmentioning

confidence: 99%

An intrusion detection system based on convolution neural network

Mo,

Li,

Wang

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

With the rapid extensive development of the Internet, users not only enjoy great convenience but also face numerous serious security problems. The increasing frequency of data breaches has made it clear that the network security situation is becoming increasingly urgent. In the realm of cybersecurity, intrusion detection plays a pivotal role in monitoring network attacks. However, the efficacy of existing solutions in detecting such intrusions remains suboptimal, perpetuating the security crisis. To address this challenge, we propose a sparse autoencoder-Bayesian optimization-convolutional neural network (SA-BO-CNN) system based on convolutional neural network (CNN). Firstly, to tackle the issue of data imbalance, we employ the SMOTE resampling function during system construction. Secondly, we enhance the system’s feature extraction capabilities by incorporating SA. Finally, we leverage BO in conjunction with CNN to enhance system accuracy. Additionally, a multi-round iteration approach is adopted to further refine detection accuracy. Experimental findings demonstrate an impressive system accuracy of 98.36%. Comparative analyses underscore the superior detection rate of the SA-BO-CNN system.

show abstract

“…[44] Permutation -Classification Permutation (bootstrap) tests for a cross-validation setup. [45] Parametric -Classification Blocked 3 × 2 cross validation estimator of variance. [5] Non-parametric -Optimisation Analysis of convergence using Page test.…”

Section: Survey On Statistical Analyses Proposedmentioning

confidence: 99%

Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: Practical guidelines and a critical review

Carrasco¹,

García²,

Rueda³

et al. 2020

Swarm and Evolutionary Computation

402

View full text Add to dashboard Cite

A key aspect of the design of evolutionary and swarm intelligence algorithms is studying their performance. Statistical comparisons are also a crucial part which allows for reliable conclusions to be drawn. In the present paper we gather and examine the approaches taken from different perspectives to summarise the assumptions made by these statistical tests, the conclusions reached and the steps followed to perform them correctly. In this paper, we conduct a survey on the current trends of the proposals of statistical analyses for the comparison of algorithms of computational intelligence and include a description of the statistical background of these tests. We illustrate the use of the most common tests in the context of the Competition on single-objective real parameter optimisation of the IEEE Congress on Evolutionary Computation (CEC) 2017 and describe the main advantages and drawbacks of the use of each kind of test and put forward some recommendations concerning their use.According to Pesarin [63], P is a non-parametric family of distributions if it is not possible to find a finite-dimensional space Θ in which there is a one-to-one relationship between Θ and P. This means that we do not have to assume that the underlying distribution belongs to a known family of distributions. Consequently, the prerequisites for non-parametric tests such as symmetry or 9

show abstract

Blocked 3×2 Cross-Validated t-Test for Comparing Supervised Classification Learning Algorithms

Cited by 22 publications

References 10 publications

Interpretable prediction of brain activity during conversations from multimodal behavioral signals

Interpretable prediction of brain activity during conversations from multimodal behavioral signals

An intrusion detection system based on convolution neural network

Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: Practical guidelines and a critical review

Contact Info

Product

Resources

About