PMLB v1.0: An open source dataset collection for benchmarking machine learning methods

Romano, Joseph D.; Le, Trang T.; Cava, William La; Gregg, John T.; Goldberg, Daniel J.; Ray, Natasha L.; Chakraborty, Praneel; Himmelstein, Daniel; Fu, Weixuan; Moore, Jason H.

doi:10.48550/arxiv.2012.00058

Cited by 10 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Flags [20] and Ask Ubuntu [2]. We provide detailed comparative results on both the training and test accuracy of all models, and further report the model sizes, runtimes and optimality gaps of our MIP and PBO models.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Tule¹,

Lan²,

Say³

2021

Preprint

View full text Add to dashboard Cite

In this paper, we explore model-based approach to training robust and interpretable binarized regression models for multiclass classification tasks using Mixed-Integer Programming (MIP). Our MIP model balances the optimization of prediction margin and model size by using a weighted objective that: minimizes the total margin of incorrectly classified training instances, maximizes the total margin of correctly classified training instances, and maximizes the overall model regularization. We conduct two sets of experiments to test the classification accuracy of our MIP model over standard and corrupted versions of multiple classification datasets, respectively. In the first set of experiments, we show that our MIP model outperforms an equivalent Pseudo-Boolean Optimization (PBO) model and achieves competitive results to Logistic Regression (LR) and Gradient Descent (GD) in terms of classification accuracy over the standard datasets. In the second set of experiments, we show that our MIP model outperforms the other models (i.e., GD and LR) in terms of classification accuracy over majority of the corrupted datasets. Finally, we visually demonstrate the interpretability of our MIP model in terms of its learned parameters over the MNIST dataset. Overall, we show the effectiveness of training robust and interpretable binarized regression models using MIP.

show abstract

Section: Resultsmentioning

confidence: 99%

“…We trained our models over 20, 60, 100, 200, 300, 500, 700 and 1000 training instances, and reported test accuracy results over the remaining dataset. The Flags [20] dataset consists of 43 features and 5 classes with 143 instances. The dataset describes the attributes of the flags of various countries.…”

Section: Experimental Domains and Setupmentioning

confidence: 99%

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Tule¹,

Lan²,

Say³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…All the tests described in this section are based on ten publicly available data sets from the UCI Machine Learning Repository [45] and PMLB (Penn Machine Learning Benchmarks) [46]. The data we use for the tests come from different domains and have been selected ensuring a wide diversity in terms of data size, dimensionality, and number of classes.…”

Section: Rule Set Evaluationmentioning

confidence: 99%

Visual Exploration of Machine Learning Model Behavior with Hierarchical Surrogate Rule Sets

Yuan¹,

Barr²,

Overton³

et al. 2022

Preprint

View full text Add to dashboard Cite

One of the potential solutions for model interpretation is to train a surrogate model: a more transparent model that approximates the behavior of the model to be explained. Typically, classification rules or decision trees are used due to the intelligibility of their logic-based expressions. However, decision trees can grow too deep and rule sets can become too large to approximate a complex model. Unlike paths on a decision tree that must share ancestor nodes (conditions), rules are more flexible. However, the unstructured visual representation of rules makes it hard to make inferences across rules. To address these issues, we present a workflow that includes novel algorithmic and interactive solutions. First, we present Hierarchical Surrogate Rules (HSR), an algorithm that generates hierarchical rules based on user-defined parameters. We also contribute SURE, a visual analytics (VA) system that integrates HSR and interactive surrogate rule visualizations. Particularly, we present a novel feature-aligned tree to overcome the shortcomings of existing rule visualizations. We evaluate the algorithm in terms of parameter sensitivity, time performance, and comparison with surrogate decision trees and find that it scales reasonably well and outperforms decision trees in many respects. We also evaluate the visualization and the VA system by a usability study with 24 volunteers and an observational study with 7 domain experts. Our investigation shows that the participants can use feature-aligned trees to perform non-trivial tasks with very high accuracy. We also discuss many interesting observations that can be useful for future research on designing effective rule-based VA systems.

show abstract

“…In order to establish common datasets, we extended PMLB, a repository of standardized regression and classification problems [13,36], by adding 130 SR datasets with known model forms. PMLB provides utilities for fetching and handling data, recording and visualizing dataset metadata, and contributing new datasets.…”

Section: Srbenchmentioning

confidence: 99%

“…Each dataset is stored using Git Large File Storage and PMLB is planned for long-term maintenance. PMLB is available under an MIT license, and is described in detail in Romano et al [36]. The authors bear all responsibility in case of violation of rights.…”

Section: A4 Additional Dataset Informationmentioning

confidence: 99%

Contemporary Symbolic Regression Methods and their Relative Performance

La Cava,

Orzechowski,

Burlacu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Many promising approaches to symbolic regression have been presented in recent years, yet progress in the field continues to suffer from a lack of uniform, robust, and transparent benchmarking standards. In this paper, we address this shortcoming by introducing an open-source, reproducible benchmarking platform for symbolic regression. We assess 14 symbolic regression methods and 7 machine learning methods on a set of 252 diverse regression problems. Our assessment includes both real-world datasets with no known model form as well as ground-truth benchmark problems, including physics equations and systems of ordinary differential equations. For the real-world datasets, we benchmark the ability of each method to learn models with low error and low complexity relative to state-of-the-art machine learning methods. For the synthetic problems, we assess each method's ability to find exact solutions in the presence of varying levels of noise. Under these controlled experiments, we conclude that the best performing methods for real-world regression combine genetic algorithms with parameter estimation and/or semantic search drivers. When tasked with recovering exact equations in the presence of noise, we find that deep learning and genetic algorithm-based approaches perform similarly. We provide a detailed guide to reproducing this experiment and contributing new methods, and encourage other researchers to collaborate with us on a common and living symbolic regression benchmark.

show abstract

PMLB v1.0: An open source dataset collection for benchmarking machine learning methods

Cited by 10 publications

References 0 publications

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Visual Exploration of Machine Learning Model Behavior with Hierarchical Surrogate Rule Sets

Contemporary Symbolic Regression Methods and their Relative Performance

Contact Info

Product

Resources

About