Towards UCI+: A mindful repository design

Macií, Núria; Bernadó-Mansilla, Ester

doi:10.1016/j.ins.2013.08.059

Cited by 56 publications

(46 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The repository is publicly available and is regularly used in machine learning research. The usage procedure, which is referred as "The UCI test" [118] or the "de facto approach" [44] [96], follows the general form of equation 1 where M is the repository, p is the choice of datasets and R is one particular performance metric (accuracy, AUC, Brier score, F-measure, MSE, etc. [53,76]).…”

Section: Pedestrian Benchmarksmentioning

confidence: 99%

“…It is a distortion-based generator (similar to Soares's UCI++ [153]). Finally, [118] suggest ideas about sharing and arranging the results of previous evaluations so that each new algorithm can be compared immediately with many other algorithms using the same experimental setting. This idea of 'experiment database' [168] has already been set up.…”

Section: Pedestrian Benchmarksmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Hernández-Orallo

2016

Artif Intell Rev

107

View full text Add to dashboard Cite

The evaluation of artificial intelligence systems and components is crucial for the progress of the discipline. In this paper we describe and critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems. We first focus on the traditional task-oriented evaluation approach. We identify three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation. We describe some of the limitations of the many evaluation schemes and competitions in these three categories, and follow the progression of some of these tests. We then focus on a less customary (and challenging) ability-oriented evaluation approach, where a system is characterised by its (cognitive) abilities, rather than by the tasks it is designed to solve. We discuss several possibilities: the adaptation of cognitive tests used for humans and animals, the development of tests derived from algorithmic information theory or more integrated approaches under the perspective of universal psychometrics. We analyse some evaluation tests from AI that are better positioned for an ability-oriented evaluation and discuss how their problems and limitations can possibly be addressed with some of the tools and ideas that appear within the paper. Finally, we enumerate a series of lessons learnt and generic guidelines to be used when an AI evaluation scheme is under consideration.

show abstract

Section: Pedestrian Benchmarksmentioning

confidence: 99%

Section: Pedestrian Benchmarksmentioning

confidence: 99%

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Hernández-Orallo

2016

Artif Intell Rev

107

View full text Add to dashboard Cite

show abstract

“…As most of them are real datasets, it is not possible to assert that they are noiseless, although some of them are artificial and show no label inconsistency. Nonetheless, a recent study showed that most of the datasets from UCI can be regarded as easy problems, once many classification techniques are able to attain high accuracies when applied to them [32]. Table 2 summarizes the main characteristics of these datasets: number of examples (Examples), number of features (Features), number of classes (Class) and percentage of the examples in the majority class (%MC).…”

Section: Datasetsmentioning

confidence: 99%

Effect of label noise in the complexity of classification problems

2015

View full text Add to dashboard Cite

“…Therefore, the noise rates artificially added could not match to the rate of noise present in the data. The predictive performance of a classifier for a particular dataset is often associated with the difficulty of the classification problem represented by this dataset (Lorena et al, 2012;Maciá & Bernadó-Mansilla, 2014). It is intuitive that for easy classification problems it is also easy to obtain a plausible and highly accurate classification hypothesis, while the opposite is verified for difficult problems.…”

Section: Methodsmentioning

confidence: 99%

“…Because they are real, it is not possible to assert that they are noisefree, although some of them are artificial and show no label inconsistencies. Nonetheless, a recent study showed that most of the datasets from UCI can be considered easy problems, once many classification techniques are able to obtain high predictive accuracies when applied to them (Maciá & Bernadó-Mansilla, 2014). Table 2.2 summarizes the main characteristics of the datasets used in the experiments of this Thesis: number of examples (#EX), number of features (#FT), number of classes (#CL) and percentage of the examples in the majority class (%MC).…”

Section: Datasetsmentioning

confidence: 99%

Noise detection in classification problems

García¹,

Lorena²,

Carvalho³

View full text Add to dashboard Cite

Firstly, I would like to express my deep gratitude to Prof. André de Carvalho and Ana Lorena, my research supervisors. Prof. André de Carvalho is one of the few fascinating people who we have the pleasure to meet in life. An exceptional professional and a humble human being. Prof. Ana Lorena is responsible for one of the most important achievements of my life, which was the finishing of this work. She enlightened every step of this journey with her personal and professional advices. I thank both for granting me the opportunity to grow as a researcher. Besides my advisors, I would like to thank Francisco Herrera and Stan Matwin for sharing their valuable knowledge and advice during the internships. I am also thankful to Prof. João Rosa, Prof. Rodrigo Mello and Prof. Gustavo Batista for being my professors in the first half of the doctorate. With them I had the pleasure to learn the meaning of being a good professor. I thank my friends and labmates who supported me in so many different ways. To Jáder Breda, Carlos Breda, Luiz Trondoli e Alexandre Vaz for being my brothers since 2005 and expend so many coffee with me. To Davi Santos for the opportunity to know a bit of your thoughts. To Henrique Marques for all kilometers running and all breathless talks.

show abstract

Towards UCI+: A mindful repository design

Cited by 56 publications

References 22 publications

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Effect of label noise in the complexity of classification problems

Noise detection in classification problems

Contact Info

Product

Resources

About