Generalization of the maximum capacity of recurrent neural networks

Abstract

In our previous work, we have proposed a novel model, which presents the maximum capacity of 1-layer recurrent neural networks by using an initiator, A, to construct the weight matrix and threshold and to define an equation, which produces ail memorized vectors. In this paper, we generalize that model by lifting the restriction of A and give the new version of our model.Besides the explanation of new version of that model, we give more discussion about it. We also compare our model with the SOR method.

PAC-Bayesian theory for stochastic LTI systems

“…Theorem 4.1: Let M π denote the set of all absolutely continuous densities w.r.t π. Then for any density π on hypothesis class F , any δ ∈ (0, 1], and (16) the following inequality holds with probability at least 1−2δ…”

“…E[(L(f ) − V N (f )) r ].Then using Lemma A 15. we getE e λ(L(f )−VN (f )) )4(r − 1)G e (f ) 2r (A.153)Now using Lemma A 16. we obtainE e λ(L(f )−VN (f )) ≤ 1 + 1 N ∞ r=2 (m + r − 1)!…”

“…[(L νi (f ) − V N,νi (f )) r ] (A.149) From Lemma A.14, we have E[(L ν (f ) − V N,ν (f )) r ] ≤ 1 N σ(r)4(r − 1)G e (f ) 2r , thus E[(L(f ) − V N (f )) r ] ≤ )4(r − 1)G e (f ) 2r (A.150) = n r y N σ(r)4(r − 1)G e (f ) 2r (A.151)Lemma A 16…”