Detecting intraday financial market states using temporal clustering

Abstract

We propose the application of a high-speed maximum likelihood clustering algorithm to detect temporal financial market states, using correlation matrices estimated from intraday market microstructure features. We first determine the ex-ante intraday temporal cluster configurations to identify market states, and then study the identified temporal state features to extract state signature vectors which enable online state detection. The state signature vectors serve as low-dimensional state descriptors which can be used in learning algorithms for optimal planning in the high-frequency trading domain. We present a feasible scheme for real-time intraday state detection from streaming market data feeds. This study identifies an interesting hierarchy of system behaviour which motivates the need for timescale specific state space reduction for participating agents.

Fast Super-Paramagnetic Clustering

“…The so called "Louvain" algorithm [2] is agglomerative and implements the later bottom-up approach to "community detection" on networks. It is in spirit very similar to 3 Here ns = N i=1 δs i ,s, cs = N i=1 N j=1 C ij δs i ,sδs j ,s, and gs = cs−ns n 2 s −ns [3,5]. 4 Let y i = x i − gs i ηs i + 1 − g 2 s i i , and δ(y) a Dirac delta function of y which is 1 when y = 0, and 0 otherwise.…”

“…The method proposed in [5,6,15] allows for all sorts of mutations but can be sensitive to initial conditions: At every step a new generation of individuals is mutated, evaluated, and a group of the best candidates survives until the next algorithm's iteration. It has its disadvantages which are discussed in Table I: I1 Convergence Criteria: Assuming the existence of multiple local maxima it tries to navigate around these "sub-optimal" solutions on its way to a global maximum.…”

“…The Likelihood evaluation itself is inexpensive but multi-processing adds CPU-overhead. This can be mitigated by using GPUs as in [5]. Towards building a fast bottom-up merging algorithm we again start with all N spins in N cluster but we iteratively merge clusters in a greedy fashion.…”

“…Concretely, that the proposed algorithm does in fact recover the correct superparamagnetic cluster configurations that are near the entropy maxima. Previous cases studies include data clustering of stocks [15] and gene data in [4], temporal states of financial markets [8], and state-detection for adaptive machine learning in trading [5]. There is an endless variety of potential use-cases for this type of fast big-data clustering technology.…”

“…We call the new algorithm Agglomerative Super-Paramagnetic Clustering (ASPC) and it has the benefit of being less computationally expensive than the PGAs implemented in [5,6,15]. Here the key insight leading to the performance enhancement arises from being able serialize the algorithm into a brute-force search across * Electronic address: ylblio001@myuct.ac.za † Electronic address: tim.gebbie@uct.ac.za 1 For a review on the last 20 years on financial market correlation based data clustering see [9], and more generally on data clustering see [7].…”