The profound impact of negative power law noise on the estimation of causal behavior

Abstract

This paper demonstrates that highly correlated negative power law (neg-p) noise-noise with a PSD ∝ |f| p when p<0-and causal behavior contained in data cannot be properly separated from each other by any causal fitting or estimation technique. It then shows: (a) that this leads such techniques to generate anomalous estimates of the true causal behavior and to underreport the true fit error, and (b) that these anomalies cannot be corrected by adding modeling or increasing the data collection interval T. Further consequences of the above are also discussed. These include anomalous behavior in noise whitening, M-corner hat, and cross-correlation techniques and the nonobservability of unbiased measures of pure random error when neg-p noise is present. The paper also investigates the relationship between ergodic-like behavior-the approximate equality of time averages over a finite T and ensemble averages-and such anomalous behavior in general statistical processing techniques. It is shown that such ergodic-like behavior is a necessary condition for practical implementations of these techniques to behave like their theoretical counterparts and, furthermore, a necessary condition for ergodic-like behavior is T>>τ c , where τ c is the correlation time of the noise process involved. Finally, it is shown that this is unachievable for neg-p noise, because τ c = ∞ for such noise (unless system highpass filtering makes τ c finite for the system filtered noise variable).

The profound impact of negative power law noise on statistical estimation

“…also, when one examines the estimated 2nd-order components in x poly,2 (t) from these simulations, one finds that the deviations of the 2nd-order coefficient estimates from the true coefficient value varies much more widely than is obtained from theoretical E -averaged considerations. This is true even though the E -averaged theory presented above indicates that random variables for this 2nd-order coefficient [such as Δ(τ) 2 x(t)] should be bounded, Wss, and mean ergodic [10]. This is somewhat puzzling and has not been fully investigated by the author, but this discrepancy may be connected with variance non-ergodicity of the HP filtered variable Δ(τ) 2 x(t).…”

“…Fig. 2 shows how Δ-measures can be interpreted as either stability or data precision measures [6]- [10]. Here, let us start with M + 1 data points x(t m ′) separated by the time interval τ, where t m ′ = t 0 + m′τ for m′ = 0 to M. In Fig.…”

“…What is also well-known, but often forgotten in practice, is that the valid application of such fitting theory to these practical measured data applications requires the assumption of ergodic-like noise behavior over the finite data collection interval T. That is, one must assume that 〈…〉 T or finite time-or sample-based averages over T for a single ensemble member data set will approximate with sufficient accuracy their equivalent E ¼, or averages over of the whole ensemble of data sets [9], [10]. later, we will show that neg-p noise is not ergodiclike over any T (that is, it is not ergodic) and discuss in detail how this can cause practical fit realizations to generate highly anomalous results that are not consistent with theoretically predicted behavior.…”

“…neg-p noise has an infinite correlation time: τ c = ∞ neg-p noise is inherently non-ergodic: Ex p (t) ≠ 〈x p (t)〉 T→∞ [8]- [10]. [7].…”

“…We note that much of this paper is based on material previously published in conference papers by the author but never published in the Transactions [2], [5]- [10]. Thus, the reader is referred to these papers for more detailed descriptions and explanations.…”