Detecting randomwalks hidden in noise: Phase transition on large graphs

Abstract

We consider the problem of distinguishing between two hypotheses: that a sequence of signals on a large graph consists entirely of noise, or that it contains a realization of a random walk buried in the noise. The problem of computing the error exponent of the optimal detector is simple to formulate, but exhibits deep connections to problems known to be difficult, such as computing Lyapunov exponents of products of random matrices and the free entropy density of statistical mechanical systems. We describe these connections, and define an algorithm that efficiently computes the error exponent of the Neyman-Pearson detector. We also derive a closed-form formula, derived from a statistical mechanics-based approximation, for the error exponent on an arbitrary graph of large size. The derivation of this formula is not entirely rigorous, but it closely matches the empirical results in all our experiments. This formula explains a phase transition phenomenon in the error exponent: below a threshold SNR, the error exponent is nearly constant and near zero, indicating poor performance; above the threshold, there is rapid improvement in performance as the SNR increases. The location of the phase transition depends on the entropy rate of the random walk.

Fast path localization on graphs via multiscale Viterbi decoding

“…The term d(V * T ,V t ) is the distance between the two destinations of the true path P * and the estimateP. Now we show how to use a dynamic programming method to compute the RHS of (40). Define a subpath of length τ of a pathP = (V 1 ,V 2 , .…”

“…A closely related line of work considers the problem of detecting signals in irregular domains [34]- [39]. In particular, [40], [41] consider optimal random walk detection on a graph which is closely related to the problem of path localization. However, our problem setting considers path-signals that can be adversarial, in the sense that the proposed algorithm and theoretical bounds can be applied to worst-case path-signals.…”

Optimal hypothesis testing with combinatorial structure: Detecting random walks on graphs

“…We have used the notation ϕ(β) to hint at a connection between our problem and the problem of computing the free energy of a spin glass in statistical physics. We first explored this connection in [7], but the results were non-rigorous. Here, we present rigorous results connecting the error exponent to a closed-form expression derived using techniques from statistical physics.…”

“…One model in particular, though, has an exact solution that will be very useful to us: the random energy model (REM). We first explored the connection to the REM in [7]. In the random energy model, there are 2 N states and the Hamiltonian function is purely random: H(S) ∼ N (0, N), Using saddle point techniques, the free energy density can be found to be:…”

“…We do this by computing a lower bound to the error exponent using a connection to statistical physics. We first explored this connection in [7], but the results were non-rigorous. In this paper, we show that the lower bound is rigorous.…”