Orthogonal subsampling for big data linear regression

Wang, Lin; Elmstedt, Jake; Wong, Weng Kee; Xu, Hongquan

doi:10.1214/21-aoas1462

Cited by 30 publications

(16 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some omitted topics include social media text data [145], virus lineage [146], public health data monitoring [147] and analysis of administrative and translation data [148]. Second, analysis for big data may alternatively begin with a selected subset of the big data with some optimality properties; see [149]. Modern statistical methods are then applied to infer the key messages from the subset data to the massive data.…”

Section: Discussionmentioning

confidence: 99%

An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic

Fei,

Ryeznik,

Sverdlov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In the era of big data, standard analysis tools may be inadequate for making inference and there is a growing need for more efficient and innovative ways to collect, process, analyze and interpret the massive and complex data. We provide an overview of challenges in big data problems and describe how innovative analytical methods, machine learning tools and metaheuristics can tackle general healthcare problems with a focus on the current pandemic. In particular, we give applications of modern digital technology, statistical methods,data platforms and data integration systems to improve diagnosis and treatment of diseases in clinical research and novel epidemiologic tools to tackle infection source problems, such as finding Patient Zero in the spread of epidemics. We make the case that analyzing and interpreting big data is a very challenging task that requires a multi-disciplinary effort to continuously create more effective methodologies and powerful tools to transfer data information into knowledge that enables informed decision making.

show abstract

Section: Discussionmentioning

confidence: 99%

An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic

Fei,

Ryeznik,

Sverdlov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Such distributed covariates may not be as common as bounded or normally distributed covariates in real applications, and when covariates do have heavy-tail distributions, it is common to consider suitable transformations of the covariates before a linear model is fitted. Figure 1 in the supplementary materials (Wang et al, 2021) shows that MSE β0 does not change much across the three methods.…”

Section: The Parameter T I In the Eliminationmentioning

confidence: 97%

“…In Section 5, we examine the performance of the OSS method through extensive simulations, discuss models with interactions, and demonstrate the utility of using the OSS method to select subsamples for two real big data sets. We offer concluding remarks in Section 6 and show the proof of technical results in the supplementary materials (Wang et al, 2021).…”

mentioning

confidence: 88%

See 1 more Smart Citation

Orthogonal Subsampling for Big Data Linear Regression

Wang¹,

Elmstedt²,

Wong³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The dramatic growth of big datasets presents a new challenge to data storage and analysis. Data reduction, or subsampling, that extracts useful information from datasets is a crucial step in big data analysis. We propose an orthogonal subsampling (OSS) approach for big data with a focus on linear regression models. The approach is inspired by the fact that an orthogonal array of two levels provides the best experimental design for linear regression models in the sense that it minimizes the average variance of the estimated parameters and provides the best predictions. The merits of OSS are three-fold: (i) it is easy to implement and fast; (ii) it is suitable for distributed parallel computing and ensures the subsamples selected in different batches have no common data points; and (iii) it outperforms existing methods in minimizing the mean squared errors of the estimated parameters and maximizing the efficiencies of the selected subsamples. Theoretical results and extensive numerical results show that the OSS approach is superior to existing subsampling approaches. It is also more robust to the presence of interactions among covariates and, when they do exist, OSS provides more precise estimates of the interaction effects than existing methods. The advantages of OSS are also illustrated through analysis of real data.

show abstract

“…\end{equation}$$The algorithms we report search for a design, which minimises the mean square error (MSE) of the least square estimate (LSE) of the

\pmb {\theta }

parameters, guarding against the two different sources of bias. Recently, authors of Refs 1,2,17,18 . proposed methods of data selection from large datasets in a DoE context, as a response to the more and more frequent need to analyse Big Data.…”

Section: Model‐oriented Selection Of Sub‐datasetsmentioning

confidence: 99%

Large datasets, bias and model‐oriented optimal design of experiments

Pesce

Porro

Riccomagno

2022

Quality & Reliability Eng

View full text Add to dashboard Cite

We review recent literature that proposes to adapt ideas from classical model based optimal design of experiments to problems of data selection of large datasets. Special attention is given to bias reduction and to protection against confounders. Some new results are presented. Theoretical and computational comparisons are made. K E Y W O R D Sconfounders, large datasets, model bias, optimal experimental design INTRODUCTIONFor the analysis of big datasets, statistical methods have been developed, which use the full available dataset. For example, new methodologies developed in the context of Big Data and focussed on a 'divide-and-recombine' approach are summarised in Wang et al. 19 Other major methods address the scalability of Big Data through Bayesian inference based on a Consensus Monte Carlo algorithm 13 and sparsity assumptions. 16 In contrast, other authors argue on the advantages of inference statements based on a well-chosen subset of the large dataset. Big datasets are characterised by few key factors. While usually data can be collected in scientific studies via active or passive observation, Big Data is often collected in passive way. Rarely their collection is the result of a designed process. This generates sources of bias, which either we do not know at all or are too costly to control. Nevertheless they will affect the overall distribution of the observed variables. 3,11 Many authors in Ref. 15 argues that analysis of big dataset is effected by issues of bias and confounding, selection bias and other sampling problems (see, for example, Sharpes 14 for electronic health records). Often the causal effect of interest can only be measured on the average and great care has to be taken about the background population, for example, it is possible to consider and analyse every message on Twitter and use it to draw conclusions about the public opinion, but it is known that Twitter users are not representative of the whole population. The analysis of the full dataset might be prohibitive because of computational and time constraints. Indeed, in some cases, the analysis of the full dataset might also be not advisable. 4,6 To recall just one example, where the sample proportion of a self-reported big dataset of size 2300,000 unit has the same mean squared error as the sample proportion from a suitable simple random sample (SRS) of size 400 and a Law of Large Population has been defined in order to qualify this (see Meng 9 ).Recently, some researchers argued on the usefulness of utilising methods and ideas from design of experiment (DoE) for the analysis of big datasets, more specifically from model-based optimal experimental design. They argue that specialThis is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

show abstract

Orthogonal subsampling for big data linear regression

Cited by 30 publications

References 26 publications

An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic

An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic

Orthogonal Subsampling for Big Data Linear Regression

Large datasets, bias and model‐oriented optimal design of experiments

Contact Info

Product

Resources

About