Data Shapley Valuation for Efficient Batch Active Learning

Ghorbani, Amirata; Zou, James; Esteva, Andre

doi:10.1109/ieeeconf56349.2022.10064696

Cited by 50 publications

(81 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This data value can be of social-ecological, economical [25,26], functional and/or symbolic nature [10,26] with the purpose of adding a measurable business value [13]. Thereby, data value is determined by a multitude of value drivers [27,28], underlying theories [29][30][31][32][33], as well as frameworks [34,35].…”

Section: Data Valuation Business Capabilitymentioning

confidence: 99%

“…The rst characteristic economic comprises all theories that determine the data value based on price-quantity diagrams, cost curves and conventional hardware-oriented pricing (cost, competition, customer) [29,32,57]. In addition, game theory can be used as a data valuation theory, which can be divided into two characteristics, cooperative [30,31] and non-cooperative game theory [63,75]. The fourth characteristic decision theory summarizes approaches, e.g., analytic hierarchy process [51,67] or fair knapsack [77], that assess the data value while taking uncertainty and vagueness into account.…”

Section: Data Valuation Theorymentioning

confidence: 99%

See 1 more Smart Citation

Towards a taxonomy for business capabilities determining data value

Hafner¹,

Silva²

2023

Preprint

View full text Add to dashboard Cite

Data and its valuation become increasingly crucial for enterprises and academia, which coincides a multitude of data valuation approaches including numerous affected focus areas, dimensions, and characteristics. Therefore, this paper analyzes different approaches to determine data value from a business capability perspective according to the TOGAF standard. Specifically, this paper deals with (a) the development of a taxonomy for data valuation business capabilities (DVBC) as well as (b) the taxonomy validation by the use of existing data valuation approaches. The applied methodologies are taxonomy development techniques for information systems, which are based on a previously executed systematic literature review with a sample size of 67 articles. Further, the data valuation business capability taxonomy is validated through applying two recent data valuation approaches from academia. As a result, the taxonomy developed consists of four business capability layers, nine dimensions, and 36 characteristics. The characteristics are of exclusive or non-exclusive nature, depending on their meaningfulness, and are validated by the successful application of two data valuation approaches. Compiled findings meet both objective and subjective quality standards. With the developed DVBC taxonomy, scholars and professionals are equipped with a tool to classify and structure their data valuation endeavors. In addition, the DVBC taxonomy bridges the domains of information systems, enterprise architecture management, and data management to serve as a foundation for interdisciplinary value generation with and through data.

show abstract

Section: Data Valuation Business Capabilitymentioning

confidence: 99%

Section: Data Valuation Theorymentioning

confidence: 99%

Towards a taxonomy for business capabilities determining data value

Hafner¹,

Silva²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-source AL for NLP While AL has been studied for a variety of tasks in NLP (Siddhant and Lipton, 2018;Lowell et al, 2019;Ein-Dor et al, 2020;Shelmanov et al, 2021;Margatina et al, 2021;Yuan et al, 2022;Schröder et al, 2022;Margatina et al, 2022;Kirk et al, 2022;Zhang et al, 2022), the majority of work remains limited to settings where training data is assumed to stem from a single source. Some recent works have sought to address the issues that arise when relaxing the single-source assumption (Ghorbani et al, 2021;, though results remain primarily limited to image classification. Moreover, these works study how AL fares under the presence of corrupted training data, such as duplicating images or adding Gaussian noise, and they do not consider settings where sampling from multiple sources may be beneficial due to complementary source attributes.…”

Section: Related Workmentioning

confidence: 99%

Investigating Multi-source Active Learning for Natural Language Inference

Snijders¹,

Margatina²

2023

Preprint

View full text Add to dashboard Cite

In recent years, active learning has been successfully applied to an array of NLP tasks. However, prior work often assumes that training and test data are drawn from the same distribution. This is problematic, as in real-life settings data may stem from several sources of varying relevance and quality. We show that four popular active learning schemes fail to outperform random selection when applied to unlabelled pools comprised of multiple data sources on the task of natural language inference. We reveal that uncertainty-based strategies perform poorly due to the acquisition of collective outliers, i.e., hard-to-learn instances that hamper learning and generalization. When outliers are removed, strategies are found to recover and outperform random baselines. In further analysis, we find that collective outliers vary in form between sources, and show that hard-to-learn data is not always categorically harmful. Lastly, we leverage dataset cartography to introduce difficultystratified testing and find that different strategies are affected differently by example learnability and difficulty.

show abstract

“…Recently, training dynamics, i.e. traces of SGD or logits during training, is being used in analyzing catastrophic forgetting [27], measuring the importance of data samples on a learning task [28], large dataset analysis [29], and identifying noisy labels [30]. Toneva et al [27] use training dynamics to identify forgetting events over the course of training.…”

Section: Related Workmentioning

confidence: 99%

“…Toneva et al [27] use training dynamics to identify forgetting events over the course of training. Ghorbani et al [28] propose a method called Data Shapley to quantify the predictor performance of each training sample to identify potentially corrupted data samples. Dataset Cartography [29] analyzes a large-scale dataset map with training dynamics to identify the presence of ambiguous-, easy-, and hard-to-learn data regions in a feature space.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Deep One-Class Classification with Adaptive Threshold based on Training Dynamics

Kim

et al. 2022

2022 IEEE International Conference on Data Mining Workshops (ICDMW)

View full text Add to dashboard Cite

One-class classification has been a prevailing method in building deep anomaly detection models under the assumption that a dataset consisting of normal samples is available. In practice, however, abnormal samples are often mixed in a training dataset, and they detrimentally affect the training of deep models, which limits their applicability. For robust normality learning of deep practical models, we propose an unsupervised deep oneclass classification that learns normality from pseudo-labeled normal samples, i.e., outlier detection in single cluster scenarios. To this end, we propose a pseudo-labeling method by an adaptive threshold selected by ranking-based training dynamics. The experiments on 10 anomaly detection benchmarks show that our method effectively improves performance on anomaly detection by sizable margins.

show abstract

Data Shapley Valuation for Efficient Batch Active Learning

Cited by 50 publications

References 16 publications

Towards a taxonomy for business capabilities determining data value

Towards a taxonomy for business capabilities determining data value

Investigating Multi-source Active Learning for Natural Language Inference

Unsupervised Deep One-Class Classification with Adaptive Threshold based on Training Dynamics

Contact Info

Product

Resources

About