Large datasets: Segmentation, feature extraction, and compression

“…The research presented here is part of a project to develop new algorithms and software for identifying meaningful events in large datasets [3,4]. In generic terms, the goal of the analysis is to identify unusual events for the immediate purpose of saving those segments of the data series, making the assumption that the resulting collection of unusual events is ammenable to classical statistical analysis methods.…”

Section: Introductionmentioning

confidence: 99%

Analysing perturbations and nonstationarity in data series using techniques motiviated by the theory of chaotic nonlinear dynamical systems

Downing¹,

Fedorov²,

Lawkins³

et al. 1996

Self Cite

View full text Add to dashboard Cite

LOCKHEED MAAnN ENERGY RESEARCH CORPORATiON lCCSTRll3UTlON OF THIS OQCUMENT I S UNLIMITED MASTERThis report has been reproduced directly from the best available copy. ARM example data series with perturbations color coded by duration according to the description given in Table 1: ordinate = data(y); abscissa EEG example data series with perturbations color coded by duration according to the description given in Table 1 Large data series with more than several million multivariate observations, r e p resenting tens of megabytes or even gigabytes of data, are difficult or impossible to analyze with traditional software. The shear amount of data quickly overwhelmes both the available computing resources and the ability of the investigator to confidently identify meaningful patterns and trends which may be present. The purpose of this research is, first, to give a meaninghl definition to "large data set analysis" and, second, to described and illustrate a technique for identifying unusual events in large data series. The technique presented here is based on the theory of nonlinear dynamical systems..I

show abstract

Understanding Large‐Scale Structure in Massive Data Sets

Braverman

2008

Encyclopedia of Quantitative Risk Analysis and Assessment

View full text Add to dashboard Cite

This article discusses massive data sets and some statistical methods for analyzing them. This is not a comprehensive review of all aspects of massive data set analysis, but rather a review and discussion of the issues in the context of statistics. We start with what a “massive” data set is, and then review several analysis methods that have been proposed in the statistics literature. These have certain common threads, most notably their relationships to the concept of data reduction. Then we examine how they can be applied with the specific aim of understanding local structure on a global scale, thus exploiting one important characteristic of massive data sets. We conclude with a brief discussion of the challenges and potential benefits for risk analysis, focusing on an example from climate studies.

show abstract

Large datasets: Segmentation, feature extraction, and compression

Cited by 4 publications

References 11 publications

Large data series: modeling the usual to identify the unusual

Large data series: modeling the usual to identify the unusual

Analysing perturbations and nonstationarity in data series using techniques motiviated by the theory of chaotic nonlinear dynamical systems

Understanding Large‐Scale Structure in Massive Data Sets

Contact Info

Product

Resources

About