Dataset Distillation

Wang, Tongzhou; Zhu, Jun‐Yan; Torralba, Antonio; Efros, Alexei A.

doi:10.48550/arxiv.1811.10959

Cited by 108 publications

(231 citation statements)

References 17 publications

Supporting

Mentioning

227

Contrasting

Order By: Relevance

“…In such a case, the proposed method limits the number of trajectories to be imitated by excluding some of the various trajectories as noise, and thus it can be trained by a standard implementation. This can be interpreted as the dataset distillation [43] in the loss function stage implicitly.…”

Section: Other Applications Of the Proposed Methodsmentioning

confidence: 99%

Towards Autonomous Driving of Personal Mobility with Small and Noisy Dataset using Tsallis-statistics-based Behavioral Cloning

Kobayashi

Enomoto²

2021

Preprint

View full text Add to dashboard Cite

Autonomous driving has made great progress and been introduced in practical use step by step. On the other hand, the concept of personal mobility is also getting popular, and its autonomous driving specialized for individual drivers is expected for a new step. However, it is difficult to collect a large driving dataset, which is basically required for the learning of autonomous driving, from the individual driver of the personal mobility. In addition, when the driver is not familiar with the operation of the personal mobility, the dataset will contain non-optimal data. This study therefore focuses on an autonomous driving method for the personal mobility with such a small and noisy, so-called personal, dataset. Specifically, we introduce a new loss function based on Tsallis statistics that weights gradients depending on the original loss function and allows us to exclude noisy data in the optimization phase. In addition, we improve the visualization technique to verify whether the driver and the controller have the same region of interest. From the experimental results, we found that the conventional autonomous driving failed to drive properly due to the wrong operations in the personal dataset, and the region of interest was different from that of the driver. In contrast, the proposed method learned robustly against the errors and successfully drove automatically while paying attention to the similar region to the driver.

show abstract

Section: Other Applications Of the Proposed Methodsmentioning

confidence: 99%

Towards Autonomous Driving of Personal Mobility with Small and Noisy Dataset using Tsallis-statistics-based Behavioral Cloning

Kobayashi

Enomoto²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Also, unlike the work [13,40] which violates the rule that clients should never share data to other clients or the server, zero-shot data augmentation synthesize data based on the model information only. Note that using synthesized samples for data augmentation differs from related works like [9], which take an approach similar to dataset distillation [32] to synthesize data for the purpose of compressing model updates for communication efficiency purposes.…”

Section: Zero-shot Data Augmentationmentioning

confidence: 99%

Towards Fair Federated Learning with Zero-Shot Data Augmentation

Hao¹,

El-Khamy²,

Lee³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models, while having no access to the client data. Although it is recognized that statistical heterogeneity of the client local data yields slower global model convergence, it is less commonly recognized that it also yields a biased federated global model with a high variance of accuracy across clients. In this work, we aim to provide federated learning schemes with improved fairness. To tackle this challenge, we propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity, and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server). Empirical results on a suite of datasets demonstrate the effectiveness of our methods on simultaneously improving the test accuracy and fairness.

show abstract

“…Even though, it is non-trivial to develop a suitable framework and to design an appropriate training objective that can effectively update the distilled data towards the original data. Inspired by the previous works [1,18], we made efforts to conduct a function where the input is the distilled data and output is a well-trained model, so that the gradients can be backpropagated to the distilled data via this function when using the original data to train the model.…”

Section: Approachmentioning

confidence: 99%

Data Distillation for Text Classification

Li,

2021

Preprint

View full text Add to dashboard Cite

Deep learning techniques have achieved great success in many fields, while at the same time deep learning models are getting more complex and expensive to compute. It severely hinders the wide applications of these models. In order to alleviate this problem, model distillation emerges as an effective means to compress a large model into a smaller one without a significant drop in accuracy. In this paper, we study a related but orthogonal issue, data distillation, which aims to distill the knowledge from a large training dataset down to a smaller and synthetic one. It has the potential to address the large and growing neural network training problem based on the small dataset. We develop a novel data distillation method for text classification. We evaluate our method on eight benchmark datasets. The results that the distilled data with the size of 0.1% of the original text data achieves approximately 90% performance of the original is rather impressive. CCS CONCEPTS• Information systems → Data mining.

show abstract

Dataset Distillation

Cited by 108 publications

References 17 publications

Towards Autonomous Driving of Personal Mobility with Small and Noisy Dataset using Tsallis-statistics-based Behavioral Cloning

Towards Autonomous Driving of Personal Mobility with Small and Noisy Dataset using Tsallis-statistics-based Behavioral Cloning

Towards Fair Federated Learning with Zero-Shot Data Augmentation

Data Distillation for Text Classification

Contact Info

Product

Resources

About