Features of Big Data and sparsest solution in high confidence set

Fan, Jianqing

doi:10.1201/b16720-50

Cited by 19 publications

(12 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But in order to use them, we minimally need to know how much they can help or whether they can actually do more harm than help. The following development was built upon an earlier idea in Meng (2014), where an approximate identity was obtained because of the use of the propensity instead of the actual data recording indicator, as defined below.…”

Section: A Fundamental Identity For Data Quality-quantity Tradeoffmentioning

confidence: 99%

Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election

Meng¹

2018

Ann. Appl. Stat.

229

233

View full text Add to dashboard Cite

Statisticians are increasingly posed with thought-provoking and even paradoxical questions, challenging our qualifications for entering the statistical paradises created by Big Data. By developing measures for data quality, this article suggests a framework to address such a question: "Which one should I trust more: a 1% survey with 60% response rate or a self-reported administrative dataset covering 80% of the population?" A 5-element Eulerformula-like identity shows that for any dataset of size n, probabilistic or not, the difference between the sample average X n and the population average X N is the product of three terms: (1) a data quality measure, ρ R,X , the correlation between X j and the response/recording indicator R j ; (2) a data quantity measure, √ (N − n)/n, where N is the population size; and (3) a problem difficulty measure, σ X , the standard deviation of X. This decomposition provides multiple insights: (I) Probabilistic sampling ensures high data quality by controlling ρ R,X at the level of N −1/2 ; (II) When we lose this control, the impact of N is no longer canceled by ρ R,X , leading to a Law of Large Populations (LLP), that is, our estimation error, relative to the benchmarking rate 1/ √ n, increases with √ N; and (III) the "bigness" of such Big Data (for population inferences) should be measured by the relative size f = n/N , not the absolute size n; (IV) When combining data sources for population inferences, those relatively tiny but higher quality ones should be given far more weights than suggested by their sizes.Estimates obtained from the Cooperative Congressional Election Study (CCES) of the 2016 US presidential election suggest a ρ R,X ≈ −0.005 for self-reporting to vote for Donald Trump. Because of LLP, this seemingly minuscule data defect correlation implies that the simple sample proportion of the self-reported voting preference for Trump from 1% of the US eligible voters, that is, n ≈ 2,300,000, has the same mean squared error as the corresponding sample proportion from a genuine simple random sample of size n ≈ 400, a 99.98% reduction of sample size (and hence our confidence). The

show abstract

Section: A Fundamental Identity For Data Quality-quantity Tradeoffmentioning

confidence: 99%

Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election

Meng¹

2018

Ann. Appl. Stat.

229

233

View full text Add to dashboard Cite

show abstract

“…Meng in [19] has proposed the statistical issues posed by multi-resolution as an important statistical research area. Here we focus on a comparison with multi-resolution theory as recently described in a recent paper by Gavish, Nadler and Coifman [10].…”

Section: Comparison With Multi-resolution Theorymentioning

confidence: 99%

Inference of a Dyadic Measure and Its Simplicial Geometry from Binary Feature Data and Application to Data Quality

Ness

2019

Association for Women in Mathematics Series

View full text Add to dashboard Cite

“…the posterior membership predictor of LSM, has good large sample properties, the MLE of cluster specific ERGM parameters conditioned on it should does. However, the prob-lem is that those two models, HERGM and LSM, may be uncongenial to each other, meaning that no model can be compatible with both of them [18]. Apparently, they make very different assumptions about data, as in ERGM they follow an exponential family and in LSM they are conditional independent, as well as the ERGM MLE is a frequentist's procedure and LSM is of Bayesian.…”

Section: ∂ ∂θmentioning

confidence: 99%

A Two-Stage Working Model Strategy for Network Analysis Under Hierarchical Exponential Random Graph Models

Cao

Chen

Fujimoto

et al. 2018

2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)

View full text Add to dashboard Cite

Social networks as a representation of relational data, often possess multiple types of dependency structures at the same time. There could be clustering (beyond homophily) at a macro level as well as transitivity (a friend's friend is more likely to be also a friend) at a micro level. Motivated by [21] which constructed a family of Exponential Random Graph Models (ERGM) with local dependence assumption, we argue that this kind of hierarchical models has potential to better fit real networks. To tackle the non-scalable estimation problem, the cost paid for modeling power, we propose a two-stage working model strategy that first utilize Latent Space Models (LSM) for their strength on clustering, and then further tune ERGM to archive goodness of fit.for instance, if i cites j and j cites k, then it is more likely that i also cites k. However, we should also expect different strength of transitivity among statisticians working in different areas (clustering). Figure 1 is a simulated (undirected) network of three communities each has the same number of nodes (20) but with different transitivities, while any probability of a between-cluster tie is the same (0.05). Using the R package latentnet [15], we can see the structures clearly: the cluster with high transitivity (blue) has its nodes closer to each other, and the one with low transitivity (red) is more spreading. The uncertainties of clustering are indicated by the pie of colors.When the network is homogeneous, i.e. has one single community, Exponential Random Graph Models (ERGM) is a popular tool for modeling as it provides researchers an intuitive formulation of related structures to test various social theories [28]. The Monte Caolo Markov Chain (MCMC) techniques developed in [10,12,22] and computer programs statnet [6] led to its widespread use. However, practitioners often found that the programs had convergence problems for many specifications, and the statistical properties of the MLE are not comprehensive. Recently, [21] suggests that the distribution of sufficient statistics in the traditional ERGM can be asymptotically normal if some local dependence is imposed. Hence solves the notorious degeneracy problem, which is mainly caused by the global dependence introduced by the Markov property [2]. However, the Bayesian inference procedure proposed by their paper is extremely expensive in computation as it involves two exponentially increasing terms, one nested in the other. Present work. Motivated by the Hierarchical ERGM construction in [21], we attempt to find a general and feasible procedure to infer clustering and the within-cluster dependences simultaneousy. However, our purpose is not to archive the large sample properties as the number of clusters goes to ∞ in a Frequentist way, because we view the network as a fixed set of nodes and the parameter estimates not only for interpretation but also as a parsimonious local mechanism to represent global structure, e.g. improving model goodness of fit [11]. To illustrate our ideas more efficiently, from here...

show abstract

Features of Big Data and sparsest solution in high confidence set

Cited by 19 publications

References 42 publications

Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election

Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election

Inference of a Dyadic Measure and Its Simplicial Geometry from Binary Feature Data and Application to Data Quality

A Two-Stage Working Model Strategy for Network Analysis Under Hierarchical Exponential Random Graph Models

Contact Info

Product

Resources

About