Inference in OSNs via Lightweight Partial Crawls

Abstract

Are Online Social Network (OSN) A users more likely to form friendships with those with similar attributes? Do users at an OSN B score content more favorably than OSN C users? Such questions frequently arise in the context of Social Network Analysis (SNA) but often crawling an OSN network via its Application Programming Interface (API) is the only way to gather data from a third party. To date, these partial API crawls are the majority of public datasets and the synonym of lack of statistical guarantees in incompletedata comparisons, severely limiting SNA research progress. Using regenerative properties of the random walks, we propose estimation techniques based on short crawls that have proven statistical guarantees. Moreover, our short crawls can be implemented in massively distributed algorithms. We also provide an adaptive crawler that makes our method parameter-free, significantly improving our statistical guarantees. We then derive the Bayesian approximation of the posterior of the estimates, and in addition, obtain an estimator for the expected value of node and edge statistics in an equivalent configuration model or Chung-Lu random graph model of the given network (where nodes are connected randomly) and use it as a basis for testing null hypotheses. The theoretical results are supported with simulations on a variety of real-world networks.

Non-Markovian Monte Carlo on Directed Graphs

“…The numbers of nodes and edges in the largest strongly connected component (LSCC) of each graph are given in the fourth and fifth The primary focus here is, through numerical simulations, to confirm that the historical empirical distributionμ t by our NMMC method in Algorithms 1 and 2 (the original one and its dynamic extension) converges to the target QSD ν =π (or ν = x) and to evaluate the speed of convergence of µ t to ν . As a performance metric, we use the total variation distance (TVD) 5 between the historical empirical distributionμ t and the target QSD ν , which is given by…”

“…where N (0, σ 2 ) is a Gaussian random variable with zero mean and variance σ 2 . The (asymptotic) variance σ 2 is given by [5,44,47]…”

“…In view of the Perron-Frobenius theorem, π is the only positive eigenvector ofP and is the leading left eigenvector, corresponding to the maximal eigenvalue 1/c. On the other hand, as explained in Section 3.1, from the Perron-Frobenius theorem we have the existence of a unique QSD ν of the transient chain P that verifies νP = λν , with λ in (5). Therefore, the unique QSD ν must coincide with our desired probability distribution π, with λ = 1/c ∈ (0, 1).…”

“…To remedy this possibly slow diffusion problem, we observe that the target QSD ν = π when achieving any desired distribution π is completely controllable to our benefit. To be precise, if it is of primary concern to estimate E π { f } = i ∈N f (i)π (i) for a given sampling function f : N → R, which is often the case for 'undirected' graph sampling [5,32,37,44,58,65], the target distribution can be controlled together with a so-called importance sampling method in MCMC [47]. With an alternative sampling distribution π ′ , it is to estimate…”

“…The sampling problem with such limited access to the space often arises when sampling and estimating structural properties of various types of large networks, including peer-to-peer networks and online social networks, whose complete information is generally unavailable in advance. MCMC has then been a key enabler to obtain a sequence of samples by constructing an ergodic Markov chain or 'random walk' on a given graph, whose equilibrium distribution equals the target distribution, with only a limited and local knowledge of the graph structure [5,32,37,44,45,58,65,[76][77][78][79].…”