Nonparametric Statistical Analysis of Machine Learning Algorithms for Regression Problems

Graczyk, Magdalena; Lasota, Tadeusz; Telec, Zbigniew; Trawiński, Bogdan

doi:10.1007/978-3-642-15387-7_15

Cited by 24 publications

(24 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In our case study we applied 6 regression neural algorithms to 29 benchmark data sets and performed a statistical analysis of the results obtained using nonparametric tests and post-hoc procedures designed especially for multiple comparisons. Our preliminary work was presented at the KES2010 conference (Graczyk et al, 2010).…”

Section: B Trawiński Et Almentioning

confidence: 99%

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

Trawiński¹,

Smźtek²,

Telec³

et al. 2012

International Journal of Applied Mathematics and Computer Science

Self Cite

139

View full text Add to dashboard Cite

In the paper we present some guidelines for the application of nonparametric statistical tests and post-hoc procedures devised to perform multiple comparisons of machine learning algorithms. We emphasize that it is necessary to distinguish between pairwise and multiple comparison tests. We show that the pairwise Wilcoxon test, when employed to multiple comparisons, will lead to overoptimistic conclusions. We carry out intensive normality examination employing ten different tests showing that the output of machine learning algorithms for regression problems does not satisfy normality requirements. We conduct experiments on nonparametric statistical tests and post-hoc procedures designed for multiple 1 × N and N × N comparisons with six different neural regression algorithms over 29 benchmark regression data sets. Our investigation proves the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.

show abstract

Section: B Trawiński Et Almentioning

confidence: 99%

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

Trawiński¹,

Smźtek²,

Telec³

et al. 2012

International Journal of Applied Mathematics and Computer Science

Self Cite

139

View full text Add to dashboard Cite

show abstract

“…independence, normality, and heteroscedasticity. This is not the case in the majority of experiments in machine learning [37]. Thus, we investigated the statistical significance of the differences on performance using the nonparametric Wilcoxon test; we kept the result of the AUC measure for each fold and each classifier, and then compared them using Wilcoxon [17].…”

Section: Methodsmentioning

confidence: 99%

Feature-based classification of human transcription factors into hypothetical sub-classes related to regulatory function

2016

View full text Add to dashboard Cite

BackgroundTranscription factors are key proteins in the regulation of gene transcription. An important step in this process is the opening of chromatin in order to make genomic regions available for transcription. Data on DNase I hypersensitivity has previously been used to label a subset of transcription factors as Pioneers, Settlers and Migrants to describe their potential role in this process. These labels represent an interesting hypothesis on gene regulation and possibly a useful approach for data analysis, and therefore we wanted to expand the set of labeled transcription factors to include as many known factors as possible. We have used a well-annotated dataset of 1175 transcription factors as input to supervised machine learning methods, using the subset with previously assigned labels as training set. We then used the final classifier to label the additional transcription factors according to their potential role as Pioneers, Settlers and Migrants. The full set of labeled transcription factors was used to investigate associated properties and functions of each class, including an analysis of interaction data for transcription factors based on DNA co-binding and protein-protein interactions. We also used the assigned labels to analyze a previously published set of gene lists associated with a time course experiment on cell differentiation.ResultsThe analysis showed that the classification of transcription factors with respect to their potential role in chromatin opening largely was determined by how they bind to DNA. Each subclass of transcription factors was enriched for properties that seemed to characterize the subclass relative to its role in gene regulation, with very general functions for Pioneers, whereas Migrants to a larger extent were associated with specific processes. Further analysis showed that the expanded classification is a useful resource for analyzing other datasets on transcription factors with respect to their potential role in gene regulation. The analysis of transcription factor interaction data showed complementary differences between the subclasses, where transcription factors labeled as Pioneers often interact with other transcription factors through DNA co-binding, whereas Migrants to a larger extent use protein-protein interactions. The analysis of time course data on cell differentiation indicated a shift in the regulatory program associated with Pioneer-like transcription factors during differentiation.ConclusionsThe expanded classification is an interesting resource for analyzing data on gene regulation, as illustrated here on transcription factor interaction data and data from a time course experiment. The potential regulatory function of transcription factors seems largely to be determined by how they bind DNA, but is also influenced by how they interact with each other through cooperativity and protein-protein interactions.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-016-1349-2) contains supplementary material, which is available...

show abstract

“…Hence, the modified kernel regression for function approximation is proposed to enhance and improve the original NadarayaWatson kernel regression. Whereas the existing techniques to solve small samples rely on the ANNBP, a non-deterministic prediction model [19] that tends to produce inconsistent predictions, the proposed model produces consistent predictions due to the convexity of the weight estimation during training. The proposed model also does not require an artificial sample generation method to be incorporated in the model development.…”

Section: Introductionmentioning

confidence: 96%

Enhanced Nadaraya-Watson Kernel Regression: Surface Approximation for Extremely Small Samples

Shapiai

Ibrahim

Khalid

et al. 2011

2011 Fifth Asia Modelling Symposium

View full text Add to dashboard Cite

The function approximation problem is to find the appropriate relationship between a dependent and independent variable(s). Function approximation algorithms generally require sufficient samples to approximate a function. Insufficient samples may cause any approximation algorithm to result in unsatisfactory predictions. To solve this problem, a function approximation algorithm called Weighted Kernel Regression (WKR), which is based on Nadaraya-Watson kernel regression, is proposed. In the proposed framework, the original NadarayaWatson kernel regression algorithm is enhanced by expressing the observed samples in a square kernel matrix. The WKR is trained to estimate the weight for the testing phase. The weight is estimated iteratively and is governed by the error function to find a good approximation model. Two experiments are conducted to show the capability of the WKR. The results show that the proposed WKR model is effective in cases where the target surface function is non-linear and the given training sample is small. The performance of the WKR is also compared with other existing function approximation algorithms, such as artificial neural networks (ANN).

show abstract

Nonparametric Statistical Analysis of Machine Learning Algorithms for Regression Problems

Cited by 24 publications

References 20 publications

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

Feature-based classification of human transcription factors into hypothetical sub-classes related to regulatory function

Enhanced Nadaraya-Watson Kernel Regression: Surface Approximation for Extremely Small Samples

Contact Info

Product

Resources

About