We study the large N saddle points of two SYK chains coupled by an interaction that is nonlocal in Euclidean time. We start from analytic treatment of the free case with q = 2 and perform the numerical study of the interacting case q = 4. We show that in both cases there is a nontrivial phase structure with infinite number of phases. Every phase correspond to a saddle point in the non-interacting two-replica SYK. The nontrivial saddle points have non-zero value of the replica-nondiagonal correlator in the sense of quasi-averaging, when the coupling between replicas is turned off. Thus, the nonlocal interaction between replicas provides a protocol for turning the nonperturbatively subleading effects in SYK into non-equilibrium configurations which dominate at large N . For comparison we also study two SYK chains with local interaction for q = 2 and q = 4. We show that the q = 2 model also has a similar phase structure, whereas in the q = 4 model, dual to the traversable wormhole, the phase structure is different. arXiv:1905.04203v1 [hep-th]