AI and the Everything in the Whole Wide World Benchmark

Raji, Inioluwa Deborah; Bender, Emily M.; Paullada, Amandalynne; Denton, Emily; Hanna, Alex

doi:10.48550/arxiv.2111.15366

Cited by 15 publications

(27 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Benchmarking is a way of evaluating and comparing new methods in ML for performance on a particular dataset [81]. Following Raji et al [85], we can understand a benchmark, for the purposes of this paper, as a dataset plus a metric for measuring the performance of a particular model on a specific task. For example, suppose that the current state of the art of image classification on ImageNet for top-1 accuracy is 85%.…”

Section: Benchmarking Ethical Decisionsmentioning

confidence: 99%

See 1 more Smart Citation

Moral dilemmas for moral machines

LaCroix

2022

AI Ethics

View full text Add to dashboard Cite

Autonomous systems are being developed and deployed in situations that may require some degree of ethical decision-making ability. As a result, research in machine ethics has proliferated in recent years. This work has included using moral dilemmas as validation mechanisms for implementing decision-making algorithms in ethically-loaded situations. Using trolley-style problems in the context of autonomous vehicles as a case study, I argue (1) that this is a misapplication of philosophical thought experiments because (2) it fails to appreciate the purpose of moral dilemmas, and (3) this has potentially catastrophic consequences; however, (4) there are uses of moral dilemmas in machine ethics that are appropriate and the novel situations that arise in a machine-learning context can shed some light on philosophical work in ethics.

show abstract

Section: Benchmarking Ethical Decisionsmentioning

confidence: 99%

“…However, if the new method performs better than this benchmark, it is the best-performing algorithm to date (again, modulo efficiency, the volume of training data, etc.). By and large, state-of-the-art progress on certain benchmarks is typically taken to indicate progress on a particular task or set of tasks [85].…”

Section: Benchmarking Ethical Decisionsmentioning

confidence: 99%

Moral dilemmas for moral machines

LaCroix

2022

AI Ethics

View full text Add to dashboard Cite

show abstract

“…The model can make fast inferences and predic-tions with less than 1 second required to successfully classify a heartbeat into its respective class. 3.Generality:This paper uses the definition of generality used by [19]. This defines generality as the ability of a model to provide a balanced or generalized approach towards classification within the scope of the application domain.…”

Section: Efficiencymentioning

confidence: 99%

Multiclass Heartbeat Classification using ECG Signals and Convolutional Neural Networks

Qureshi

Camara

Poorter

et al. 2022

2022 2nd International Conference on Digital Futures and Transformative Technologies (ICoDT2)

View full text Add to dashboard Cite

Given a large enough time series signal from an ECG signal, it is possible to identify and classify heartbeats not only into normal and abnormal classes but into multiple classes including but not limited to Normal beat, Paced beat, Atrial Premature beat and Ventricular flutter as originally suggested by benchmark electrocardiogram (ECG) datasets like the MIT-BIH Arrhythmia Dataset. There are multiple approaches that target ECG classifications using Machine and Deep Learning like One Class SVM, ELM, Anogan etc. These approaches require either very high computational resources, fail to classify classes apart from normal/abnormal classes or fail to classify all classes with an equivalent or near-equivalent accuracy. With these limitations in mind, this paper proposes a deep learning approach using Convolutional Neural Networks (CNNs) to classify multiple classes of heartbeats in an efficient, effective, and generalized manner. By using the MIT-BIH Arrhythmia dataset to filter and segment individual correctly structured heartbeats, we have designed a network which can be trained on different classes of heartbeats and present robust, accurate and efficient results. The class imbalance prevalent in the MIT-BIH dataset has been dealt with using Synthetic Minority Over-sampling Technique (SMOTE). The robustness of the model is increased by adding techniques of loss minimization such as dropout and early stopping. The approach gives an accuracy of approximately 96% and an extremely short time span for class prediction(classification), i.e., less than 1 second. The results are also illustrated over multiple (10) classes to exemplify the generality of the model. We have illustrated these results over multiple (10) classes to exemplify generality of the model.

show abstract

“…Frameworks like Data Statements for Natural Language Processing (Bender & Friedman, 2018), The Dataset Nutrition Label (Holland et al, 2018), Model Cards for Model Reporting (Mitchell et al, 2019), Datasheets for Datasets (Gebru et al, 2021), Closing the AI accountability gap (Raji et al, 2021), The Ethical Pipeline for Healthcare Model Development (Chen et al, 2020), and The Clinician and Dataset Shift in Artificial Intelligence (Finlayson et al, 2021) offer guidance for breaking down the data generating process into relevant component parts to identify potential dimensions on which context shift could lead to performance changes from a test benchmark to production environments. Likewise, meta-frameworks offer guidance for ensuring data documentation frameworks are useful and actionable (Heger et al, 2022).…”

Section: Contextualizing the Benchmark-production Gapmentioning

confidence: 99%

Identifying the Context Shift between Test Benchmarks and Production Data

Groh¹

2022

Preprint

View full text Add to dashboard Cite

Across a wide variety of domains, there exists a performance gap between machine learning models' accuracy on dataset benchmarks and realworld production data. Despite the careful design of static dataset benchmarks to represent the real-world, models often err when the data is out-of-distribution relative to the data the models have been trained on. We can directly measure and adjust for some aspects of distribution shift, but we cannot address sample selection bias, adversarial perturbations, and non-stationarity without knowing the data generation process. In this paper, we outline two methods for identifying changes in context that lead to distribution shifts and model prediction errors: leveraging human intuition and expert knowledge to identify firstorder contexts and developing dynamic benchmarks based on desiderata for the data generation process. Furthermore, we present two casestudies to highlight the implicit assumptions underlying applied machine learning models that tend to lead to errors when attempting to generalize beyond test benchmark datasets. By paying close attention to the role of context in each prediction task, researchers can reduce context shift errors and increase generalization performance.

show abstract

AI and the Everything in the Whole Wide World Benchmark

Cited by 15 publications

References 52 publications

Moral dilemmas for moral machines

Moral dilemmas for moral machines

Multiclass Heartbeat Classification using ECG Signals and Convolutional Neural Networks

Identifying the Context Shift between Test Benchmarks and Production Data

Contact Info

Product

Resources

About