A scalable nonparametric specification testing for massive data

Zhao, Yanyan; Zou, Changliang; Wang, Zhaojun

doi:10.1016/j.jspi.2018.09.012

Cited by 3 publications

(1 citation statement)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the key challenges is that directly applying statistical meth-ods to super-large data with conventional computing approaches is prohibitive, which calls for the development of new tools. Recently, statistical analysis and inference in the large-scale dataset have drawn much attention, and some computationally scalable methods have been proposed to reduce the computation and storage effort from various aspects of applications, such as the divide-and-conquer procedures (Battey et al, 2018;Jordan et al, 2019;Zhao et al, 2019), subsampling strategies (Kleiner et al, 2014;Wang et al, 2018) and on-line learning methods (Balakrishnan and Madigan, 2008;Schifano et al, 2016). In most of these works, one usually assumes a parametric model, typically linear or logistic regression models.…”

Section: Introductionmentioning

confidence: 99%

Model Checking in Large-Scale Dataset via Structure-Adaptive-Sampling

Han¹,

Ma²,

Ren³

et al. 2023

STAT SINICA

Self Cite

View full text Add to dashboard Cite

Lack-of-fit testing is often essential in many applications of statistical/machine learning. Despite the availability of large-scale datasets, the challenges associated with model checking when some resource budgets are limited are not yet well addressed. In this paper, we propose a design-adaptive testing procedure to check a general model when only a limited number of data observations are available. We derive an optimal sampling strategy to select a small subset from a large pool of data, Structure-Adaptive-Sampling, with which the proposed test possesses the asymptotically best power. Numerical results on both synthetic and real-world data confirm the effectiveness of the proposed method.

show abstract