Overfitting and undercomputing in machine learning

Dietterich, Tom

doi:10.1145/212094.212114

Cited by 599 publications

(302 citation statements)

References 1 publication

Supporting

Mentioning

291

Contrasting

Unclassified

Order By: Relevance

“…A model that over-fits the data is one that has been influenced by random error and noise in the training data to the extent that it does not accurately reflect the underlying phenomenon being studied 68 . Overfitting is a particular concern in models where the number of dimensions greatly outweighs the sample size and in highly complex models.…”

Section: Accepted M Manuscriptmentioning

confidence: 99%

Using neuroimaging to help predict the onset of psychosis

et al. 2017

View full text Add to dashboard Cite

The aim of this review is to assess the potential for neuroimaging measures to facilitate prediction of the onset of psychosis. Research in this field has mainly involved people at 'ultra-high risk' (UHR) of psychosis, who have a very high risk of developing a psychotic disorder within a few years of presentation to mental health services.The review details the key findings and developments in this area to date, and examines the methodological and logistical challenges associated with making A C C E P T E D M A N U S C R I P T ACCEPTED MANUSCRIPT2 predictions in an individual subject in a clinical setting. Key wordsPsychosis prediction; Ultra High-Risk of psychosis; machine learning; Support Vector Machines; multimodal neuroimaging; multicentre neuroimaging studies; graph analysis Psychosis prediction and the ultra high-risk statePsychosis describes a syndrome that includes symptoms such as hallucinations, delusions, disorganised thought, and catatonia Criteria for UHR status include attenuated psychotic symptoms and / or a brief limited intermittent psychotic episode and / or a genetically determined vulnerability, alongside deterioration in social and occupational functioning 7 8 . UHR status comes A C C E P T E D M A N U S C R I P T ACCEPTED MANUSCRIPT3 with the caveat that those who meet UHR status are selectively those who have come into contact with clinical services. This review uses the term UHR throughout.The UHR population is strikingly heterogeneous in terms of clinical outcomes.Follow-up studies 9 10 suggest that 7 years after clinical presentation, approximately a third of UHR subjects will have developed a psychotic disorder, with most transitions occurring in the first 2 years 11. Most of those who do not develop a psychotic disorder will have persistent attenuated symptoms and / or have developed another mental health disorder, whilst 14% will have recovered (see figure 1).Clinical intervention in the UHR group may reduce the likelihood of the onset of a psychotic disorder 12. However, as most UHR subjects do not develop a psychotic disorder, providing preventative treatment to all of those at risk is clinically inefficient. Identifying biomarkers that could be used to stratify the UHR group according to clinical outcome would enable the selective delivery of preventative interventions to the subgroup that would benefit the most.It is difficult to predict clinical outcomes in an UHR subject on the basis of their clinical features at presentation. Although the clinical assessment at presentation has good diagnostic validity for ruling out a future psychotic disorder (meta analytical sensitivity of UHR assessment = 0.96) 13, it has only a modest ability to rule in a future psychotic disorder (meta-analytical specificity of UHR assessment = 0.47) 13.There is thus a need to find other forms of assessment that can improve the specificity of psychosis prediction. Fusar-Poli et al. (2015) 14 and based on data by Lin et al. (2011) 10 . A C C E P T E D M A N U S C R I P T ACCEPTED MANU...

show abstract

Section: Accepted M Manuscriptmentioning

confidence: 99%

Using neuroimaging to help predict the onset of psychosis

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Before using the big data for analysis of the system's state and real-time fault detection, pre-processing, which requires the replacement of missing values, removal of incomplete rows and columns, outliers and extreme values, was done. This process of data cleaning can also involve data integration, transformation, reduction and discretization, to make the analysis fast and prevent bogus results [37]. Hence, redundant input variables such as those that were constant were removed, and missing values were replaced with zeros and by averaging the nearest neighbors' values of the missing value cells.…”

Section: Fault Detection and Identification With Annmentioning

confidence: 99%

Integrated Big Data Analytics Technique for Real-Time Prognostics, Fault Detection and Identification for Complex Systems

Ossai

2017

Infrastructures

View full text Add to dashboard Cite

Real-time prediction of the state of complex systems is vital for integrity management since it is easier to plan for asset maintenance, reduce risks associated with unplanned downtime and reduce the cost of maintenance. This study utilized a four-fold cross-validation ensemble for an Artificial Neural Network (ANN) that used Multi-Layer Perceptron (MLP) in a backward propagation technique for haul crane prognosis. Big data on components' degradation states obtained from the Supervisory Control And Data Acquisition (SCADA) systems were used to implement the study. After preprocessing the dataset, importance scoring was used to compute the Cumulative Target-component Percentage-influence (CTP) of the input variables (source components) on the output variable (the target component) at the 95.5%, 99.3%, 99.9% and 100% levels. The specific source components responsible for the CTP levels of the target component were later used for the ANN network training that followed the cross-validation ensemble technique. The cross-validation ensemble ANN technique was also compared to the classic ANN and other machining learning algorithms. Finally, the best-trained cross-validation ensemble ANN network, which was obtained at the 99.9% CTP level, was used for future estimation of the time of failure of the system to enhance planning for the expected maintenance program that will be required at such times.

show abstract

“…Thus, in the applications for high-dimensional datasets filtering methods are often combined with the simplest or fastest learners. This is due to the fact that learning parameters usually involved in choosing complex learners may make the selection process infeasible or may result in overfitting [33]. Consequently, most frequently used models have historically been naive Bayes [34], K-nearest neighbor models or decision trees, typically C4.5 [35].…”

Section: Classification Models In Evaluating Feature Selection Algorimentioning

confidence: 99%

Feature Selection Filters Based on the Permutation Test

Radivojac¹,

Obradović²,

Dunker³

et al. 2004

Machine Learning: ECML 2004

View full text Add to dashboard Cite

Abstract. We investigate the problem of supervised feature selection within the filtering framework. In our approach, applicable to the two-class problems, the feature strength is inversely proportional to the p-value of the null hypothesis that its class-conditional densities, p(X | Y = 0) and p(X | Y = 1), are identical. To estimate the p-values, we use Fisher's permutation test combined with the four simple filtering criteria in the roles of test statistics: sample mean difference, symmetric Kullback-Leibler distance, information gain, and chi-square statistic. The experimental results of our study, performed using naive Bayes classifier and support vector machines, strongly indicate that the permutation test improves the above-mentioned filters and can be used effectively when sample size is relatively small and number of features relatively large.

show abstract

Overfitting and undercomputing in machine learning

Cited by 599 publications

References 1 publication

Using neuroimaging to help predict the onset of psychosis

Using neuroimaging to help predict the onset of psychosis

Integrated Big Data Analytics Technique for Real-Time Prognostics, Fault Detection and Identification for Complex Systems

Feature Selection Filters Based on the Permutation Test

Contact Info

Product

Resources

About