Sketching Linear Classifiers over Data Streams

Tai, Kai Sheng; Sharan, Vatsal; Bailis, Peter; Valiant, Gregory

doi:10.1145/3183713.3196930

Cited by 40 publications

(24 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We first compute the optimal detection threshold R in (6) and the corresponding failure probability pZC(R) under both scenarios, shown in Figure 1 and Figure 2 respectively. As baseline, the threshold for median filtering is (t + 1)/2 as discussed and its failure probability is p med in (5).…”

Section: Experimental Validationmentioning

confidence: 99%

See 1 more Smart Citation

Count Sketch with Zero Checking: Efficient Recovery of Heavy Components

Guanqiang

Tian

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The problem of recovering heavy components of a high-dimensional vector from compressed data is of great interest in broad applications, such as feature extraction under scarce computing memory and distributed learning under limited bandwidth. Recently, a compression algorithm called count sketch has gained wide popularity to recover heavy components in various fields. In this paper, we carefully analyze count sketch and illustrate that its default recovery method, namely median filtering, has a distinct error pattern of reporting false positives. To counteract this error pattern, we propose a new scheme called zero checking which adopts a two-step recovery approach to improve the probability of detecting false positives. Our proposed technique builds on rigorous error analysis, which enables us to optimize the selection of a key design parameter for maximum performance gain. The empirical results show that our scheme achieves better recovery accuracy than median filtering and requires less samples to accurately recover heavy components.

show abstract

Section: Experimental Validationmentioning

confidence: 99%

“…Count sketch [1], which satisfies both constraints, has been widely applied for heavy components recovery in a variety of applications such as distributed learning [3] and feature selection [4], among others [5][6][7]. Despite extensive implementations in recent years, the count sketch algorithm has rarely been examined or questioned.…”

Section: Introductionmentioning

confidence: 99%

Count Sketch with Zero Checking: Efficient Recovery of Heavy Components

Guanqiang

Tian

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Since the above three kinds of sketches are most widely used, we mainly study how to use Cluster-Reduce to compress them (see §4 for details). There are many other sketches, such as ADA-SKETCH [17], HeavyGuardian [6], WavingSketch [5], MaxLogHash [18], and others [11,14,[19][20][21][22][23], that focus on estimating item frequency, finding frequent items, measuring network traffic, estimating set similarity, and other data mining tasks.…”

Section: Sketches In Distributed Data Streamsmentioning

confidence: 99%

Cluster-Reduce

Zhao

Zhong

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Sketches, a type of probabilistic algorithms, have been widely accepted as the approximate summary of data streams. Compressing sketches is the best choice in distributed data streams to reduce communication overhead. The ideal compression algorithm should meet the following three requirements: high efficiency of compression procedure, support of direct query without decompression, and high accuracy of compressed sketches. However, no prior work can meet these requirements at the same time. Especially, the accuracy is poor after compression using existing methods. In this paper, we propose Cluster-Reduce, a framework for compressing sketches, which can meet all three requirements. Our key technique nearness clustering rearranges the adjacent counters with similar values in the sketch to significantly improve the accuracy. We use Cluster-Reduce to compress four kinds of sketches in two use-cases: distributed data streams and distributed machine learning. Extensive experimental results show that Cluster-Reduce can achieve up to 60 times smaller error than prior works. The source codes of Cluster-Reduce are available at Github anonymously [1].

show abstract

“…Another line of work that we draw from applies sketching techniques to learning tasks where the model itself cannot fit in memory [Aghazadeh et al, 2018, Tai et al, 2018. In our setting, we can afford to keep a dense version of the model in memory, and we only make use of the memory-saving properties of sketches to reduce communication between nodes participating in distributed learning.…”

Section: Related Workmentioning

confidence: 99%

Communication-efficient distributed SGD with Sketching

Ivkin¹,

Rothchild²,

Ullah³

et al. 2019

Preprint

View full text Add to dashboard Cite

Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time. Motivated by the success of sketching methods in sub-linear/streaming algorithms, we introduce SKETCHED-SGD 4 , an algorithm for carrying out distributed SGD by communicating sketches instead of full gradients. We show that SKETCHED-SGD has favorable convergence rates on several classes of functions. When considering all communication -both of gradients and of updated model weights -SKETCHED-SGD reduces the amount of communication required compared to other gradient compression methods from O(d) or O(W ) to O(log d), where d is the number of model parameters and W is the number of workers participating in training. We run experiments on a transformer model, an LSTM, and a residual network, demonstrating up to a 40x reduction in total communication cost with no loss in final model performance. We also show experimentally that SKETCHED-SGD scales to at least 256 workers without increasing communication cost or degrading model performance.

show abstract

Sketching Linear Classifiers over Data Streams

Cited by 40 publications

References 67 publications

Count Sketch with Zero Checking: Efficient Recovery of Heavy Components

Count Sketch with Zero Checking: Efficient Recovery of Heavy Components

Cluster-Reduce

Communication-efficient distributed SGD with Sketching

Contact Info

Product

Resources

About