A Unified Approach to Coreset Learning

Abstract

Coreset of a given dataset and loss function is usually a small weighed set that approximates this loss for every query from a given set of queries. Coresets have shown to be very useful in many applications. However, coresets construction is done in a problem dependent manner and it could take years to design and prove the correctness of a coreset for a specific family of queries. This could limit coresets use in practical applications. Moreover, small coresets provably do not exist for many problems. To address these limitations, we propose a generic, learning-based algorithm for construction of coresets. Our approach offers a new definition of coreset, which is a natural relaxation of the standard definition and aims at approximating the average loss of the original data over the queries. This allows us to use a learning paradigm to compute a small coreset of a given set of inputs with respect to a given loss function using a training set of queries. We derive formal guarantees for the proposed approach. Experimental evaluation on deep networks and classic machine learning problems show that our learned coresets yield comparable or even better results than the existing algorithms with worst-case theoretical guarantees (that may be too pessimistic in practice). Furthermore, our approach applied to deep network pruning provides the first coreset for a full deep network, i.e., compresses all the network at once, and not layer by layer or similar divide-and-conquer methods.

Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions

“…Coresets. In the recent years, coresets got increasing attention, and where leveraged to compress the input datasets of many machine learning algorithms, improving there performance, e.g., regression [72,38,81,47,103], decision trees [44], matrix approximation [26,70,25,89,73], data discretization [75], clustering [24,29,66,4,45,90,106], z -regression [16,17,94], SVM [32, 101, 99, 100, 102], deep learning models [69,5,60] and even for path planning in the field of robotics [104]. For extensive surveys on coresets, we refer the reader to [23,86,43,71].…”