Decoding Machine Learning Benchmarks

Cardoso, Lucas F. F.; Santos, Vitor Cirilo Araujo; Francês, Regiane S. Kawasaki; Prudêncio, Ricardo B. C.; Alves, Ronnie

doi:10.1007/978-3-030-61380-8_28

Cited by 7 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore, it is understood that OpenML has a lot to contribute to research in the field of machine learning. In the previous work (Cardoso et al, 2020) an initial analysis of OpenML-CC18 was performed using IRT, which allowed the generation of new relevant metadata about the complexity and quality of the benchmark, such as the difficulty and discriminative power of the data. In this present work, we seek to deepen this analysis by looking for a subset of datasets within OpenML-CC18 that is as good or perhaps better than the original.…”

Section: Openmlcc-18 Benchmarkmentioning

confidence: 99%

“…To build the IRT logistic models and analyze the benchmarks, the decodIRT 2 tool initially presented in Cardoso et al (2020) was used. DecodIRT has as main objective to automate the analysis of existing datasets in the OpenML platform as well as the proficiency of different classifiers.…”

Section: Decodirt Toolmentioning

confidence: 99%

“…Preliminary results were obtained in Cardoso, Santos, Francês, Prudêncio and Alves (2020), where it was possible to apply the IRT concepts to evaluate a set of 60 datasets from the OpenML-CC18 benchmark, observing the item parameters of difficulty and discrimination, using the developed tool decodIRT. This allowed us to classify the OpenML-CC18 benchmark according to its ability to separate good from bad classifiers and to test the classification power of algorithms.…”

Section: Introductionmentioning

confidence: 99%

“…Are there datasets within a benchmark that might not be really good for evaluating learning algorithms? Through this, this work aims to expand the concepts and technologies already explored initially in (Cardoso et al, 2020). For this, IRT is used to create subsets of OpenML-CC18 datasets, then the combination of IRT and Glicko-2 is applied to generate the classifier rankings.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Data vs classifiers, who wins?

Cardoso¹,

Santos²,

Francês³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The classification experiments covered by machine learning (ML) are composed by two important parts: the data and the algorithm. As they are a fundamental part of the problem, both must be considered when evaluating a model's performance against a benchmark. The best classifiers need robust benchmarks to be properly evaluated. For this, gold standard benchmarks such as OpenML-CC18 are used. However, data complexity is commonly not considered along with the model during a performance evaluation. Recent studies employ Item Response Theory (IRT) as a new approach to evaluating datasets and algorithms, capable of evaluating both simultaneously. This work presents a new evaluation methodology based on IRT and Glicko-2, jointly with the decodIRT tool developed to guide the estimation of IRT in ML. It explores the IRT as a tool to evaluate the OpenML-CC18 benchmark for its algorithmic evaluation capability and checks if there is a subset of datasets more efficient than the original benchmark. Several classifiers, from classics to ensemble, are also evaluated using the IRT models. The Glicko-2 rating system was applied together with IRT to summarize the innate ability and classifiers performance. It was noted that not all OpenML-CC18 datasets are really useful for evaluating algorithms, where only 10% were rated as being really difficult. Furthermore, it was verified the existence of a more efficient subset containing only 50% of the original size. While Randon Forest was singled out as the algorithm with the best innate ability.

show abstract

Section: Openmlcc-18 Benchmarkmentioning

confidence: 99%

Section: Decodirt Toolmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Data vs classifiers, who wins?

Cardoso¹,

Santos²,

Francês³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Diante de questões como essas, estudos recentes buscaram conhecimento em outras áreas para preencher essa lacuna. Uma nova abordagem é a aplicac ¸ão de conceitos psicométricos que são comumente utilizadas para avaliar o aprendizado de indivíduos, entre eles está a Teoria de Resposta ao Item (TRI) [Cardoso et al 2020].…”

unclassified

AutoML baseado na Teoria de Resposta ao Item

Cardoso¹,

Francês²

2022

Anais Da II Escola Regional De Alto Desempenho Norte 2 E II Escola Regional De Aprendizado De Máquina E Inteligência Artificial

View full text Add to dashboard Cite

Gerar modelos de aprendizado de máquina automaticamente segue sendo uma área de muita pesquisa recente. Porém, ainda não há uma forma definitiva capaz de gerar modelos simples que melhor generalizem e que sejam imunes a subespecificação. Neste trabalho, são apresentado os resultados preliminares de uma metodologia proposta que utiliza os conceitos da Teoria de Resposta ao Item e Algoritmo Genético para criar um algoritmo de AutoML do tipo NAS que seja capaz de gerar uma Rede Neural competitiva com os modelos já conhecidos na literatura. Nos resultados iniciais, foi possível gerar um modelo competitivo com o AutoKeras, porém com complexidade inferior a 5.5% da complexidade total do modelo do próprio AutoKeras.

show abstract