We extend the calculation of the intermediate spectral form factor (SFF) in [1] to finite N . This is the SFF of the matrix model description of U pN q Chern-Simons theory on a three-dimensional sphere, which is dual to a topological string theory characterized by string coupling gs. This matrix model displays level statistics characteristic of systems intermediate between chaotic and integrable, depending on the value of gs. We check explicitly that taking N Ñ 8 whilst keeping the string coupling gs fixed reduces the connected SFF to a pure linear ramp, thereby confirming the main result from [1] for the intermediate ensemble. We then consider the 't Hooft limit, where N Ñ 8 and gs Ñ 0 such that t " N gs remains finite. In this limit, the SFF turns into a remarkable sequence of polynomials which, as far as the authors are aware, have not appeared in the literature thus far. The two-level statistics for the intermediate ensemble in the 't Hooft limit found here may be a representative of a novel random matrix universality.