A Statistical Analysis of Probabilistic Counting Algorithms

We consider the problem of estimating the size of dynamic anonymous networks.The proposed algorithm exploits max-consensus protocols and extends a previous strategy suited for static networks. A regularization term accounts a-priori assumptions on the smoothness of the estimate, and we specifically consider quadratic regularization terms since they lead to closed-form solutions. We explicitly derive an estimation scheme tailored for peer-to-peer service networks, starting from their statistical model. Numerical experiments validate the accuracy of the algorithm and show how the strategy can be implemented using finite precision arithmetics.Problem Description dynamic network of N (t) agents each agent wants to estimate N (t) Constraints:can only use local information agents are anonymous (no global unique ID) Regularization Based Dynamic EstimationIdea: penalize implausible hypothesis ⇒ add a regularization term R to the naive approach (use a penalized log-likelihood function): Example: Peer-to-Peer Network Framework: peer-to-peer network with N max agents (either active or inactive) Assumption: birth-death process with transition probabilitiesEstimator for the 1-step memory case,

show abstract

“…Then, µ := E [N (t)] = α 1 + α N max (14) var (N (t)) = α (1 + α) 2 N max (15) cov (N (t), (16) Thus,…”

Section: A Derivation Of the Regularization Termmentioning

confidence: 99%

Distributed size estimation of dynamic anonymous networks

Terelius

Varagnolo

Johansson

2012

2012 IEEE 51st IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

show abstract

“…For two unbiased estimators using the same amount of memory, the one with a smaller relative standard error is better. For a detailed classification and comparison of existing single-flow cardinality estimation algorithms, see [10,11]. We take a closer look at the LogLog and HyperLogLog algorithms in Section 2.1.1.…”

Section: Single-flow Cardinality Estimationmentioning

confidence: 99%

Per-Flow Cardinality Estimation Based On Virtual LogLog Sketching

Zhou

Hajek

2019

2019 53rd Annual Conference on Information Sciences and Systems (CISS)

View full text Add to dashboard Cite

Flow cardinality estimation is the problem of estimating the number of distinct elements in a data flow, often with a stringent memory constraint. It has wide applications in network traffic measurement and in database systems. The virtual LogLog algorithm proposed recently by Xiao, Chen, Chen and Ling estimates the cardinalities of a large number of flows with a compact memory. The purpose of this thesis is to explore two new perspectives on the estimation process of this algorithm. Firstly, we propose and investigate a family of estimators that generalizes the original vHLL estimator and evaluate the performance of the vHLL estimator compared to other estimators in this family. Secondly, we propose an alternative solution to the estimation problem by deriving a maximum-likelihood estimator. Empirical evidence from both perspectives suggests the near-optimality of the vHLL estimator for per-flow estimation, analogous to the near-optimality of the HLL estimator for single-flow estimation.ii

show abstract

“…To bound the variance, both schemes repeat the above procedures for m different hash functions and use their combined statistics for the estimation 1 . A comprehensive overview of different cardinality estimation techniques is given in [7,21]. State-of-the art cardinality estimators have a standard error of about 1/ √ m, where m is the number of storage units [6].…”

Section: Related Work and Previous Schemesmentioning

confidence: 99%

A Minimal Variance Estimator for the Cardinality of Big Data Set Intersection

Cohen

Katzir

Yehezkel

2017

Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

In recent years there has been a growing interest in developing "streaming algorithms" for efficient processing and querying of continuous data streams. These algorithms seek to provide accurate results while minimizing the required storage and the processing time, at the price of a small inaccuracy in their output. A fundamental query of interest is the intersection size of two big data streams. This problem arises in many different application areas, such as network monitoring, database systems, data integration and information retrieval. In this paper we develop a new algorithm for this problem, based on the Maximum Likelihood (ML) method. We show that this algorithm outperforms all known schemes and that it asymptotically achieves the optimal variance. 1. For the first time, we present a complete analysis of the statistical performance (bias and variance) of the above three schemes.2. We find the optimal (minimum) variance of any unbiased set intersection estimator.3. We present and analyze a new unbiased estimator, based on the Maximum Likelihood (ML) method, which outperforms the above three schemes.The rest of the paper is organized as follows. Section 2 discusses previous work and presents the three previously known schemes. Section 3 presents our new Maximum Likelihood (ML) estimator. It also shows that the new scheme achieves optimal variance and that it outperforms the three known schemes. Section 4 analyzes the statistical performance (bias and variance) of the three known schemes. Section 5 presents simulation results confirming that the new ML estimator outperforms the three known schemes. Finally, Section 6 concludes the paper.

show abstract

A Statistical Analysis of Probabilistic Counting Algorithms

Cited by 24 publications

References 40 publications

Distributed size estimation of dynamic anonymous networks

Distributed size estimation of dynamic anonymous networks

Per-Flow Cardinality Estimation Based On Virtual LogLog Sketching

A Minimal Variance Estimator for the Cardinality of Big Data Set Intersection

Contact Info

Product

Resources

About