Universal Rank-Order Transform to Extract Signals from Noisy Data

Abstract

We introduce an ordinate method for noisy data analysis, based solely on rank information and thus insensitive to outliers. The method is nonparametric, objective, and the required data processing is parsimonious. Main ingredients are a rank-order data matrix and its transform to a stable form, which provide linear trends in excellent agreement with least squares regression, despite the loss of magnitude information. A group symmetry orthogonal decomposition of the 2D rank-order transform for iid (white) noise is further ordered by principal component analysis. This two-step procedure provides a noise "etalon" used to characterize arbitrary stationary stochastic processes. The method readily distinguishes both the Ornstein-Uhlenbeck process and chaos generated by the logistic map from white noise. Ranking within randomness differs fundamentally from that in deterministic chaos and signals, thus forming the basis for signal detection. To further illustrate the breadth of applications, we apply this ordinate method to the canonical nonlinear parameter estimation problem of two-species radioactive decay, outperforming special-purpose least square software. It is demonstrated that the method excels when extracting trends in heavy-tailed noise and, unlike the Thiele-Sen estimator, is not limited to linear regression. Lastly, a simple expression is given that yields a close approximation for signal extraction of an underlying generally nonlinear signal. arXiv:1906.08729v1 [physics.data-an]

Interaction-Assisted Reversal of Thermopower with Ultracold Atoms

“…It is known that estimating parameters of exponential models is challenging as small changes in the data can strongly influence the parameters [66]. Thus in addition to the least squares method we used a technique based on rank order that is robust against outliers and shown to improve the estimation for exponential models [67,68]. Applying this method led to comparable results and hence we conclude that for our model and data the least squares method performs well.…”

The shocklet transform: a decomposition method for the identification of local, mechanism-driven dynamics in sociotechnical time series

“…For example, Wu and Huberman described a stretched-exponential model for collective human attention [18], and Candia et al introduced a biexponential function for collective human memory on longer timescales [19]. Crane and Sornette assembled a Hawkes process for video views that produces power-law behavior by using power-law excitement kernels [20], while De Domenico and Altmann put forward a stochastic model incorporating social heterogeneity and influence [21], and Ierly and Kostinsky introduced a rank-based, signal-extraction method with applications to meteorology data [22].…”

Odd–even Decomposition of Functions

“…In fact, this note was sparked by the paper [1], which is devoted to extraction of signals from noisy data. Of particular interest to us is decomposition (14) in [1], for X = R 2 , with involutions ρ, ρ 1 , ρ 2 of R 2 given by the formulas ρ(u, v) :…”

“…More specifically, formula (14) in [1] provides a decomposition of an arbitrary function q : R 2 → R into the sum of five functions, denoted by…”

“…x , q (C 1 ) y , q (R 2 ) in [1], where: q (D 4 ) is even with respect to ρ, ρ 1 and ρ 2 ; q (D 2 ) is odd with respect to ρ, and even with respect to ρ 1 and ρ 2 ; q (C 1 )…”