Within a random matrix theory approach, we obtain spectral statistics of the Wigner time delay matrix Q, for arbitrary channels number M and for all symmetry classes, in fact for general Dyson parameter β. For small values of M , the observables develop singularities at fractional values of β which may be of physical consequence in situations of partial breaking of the time-reversal symmetry. We also put forth two conjectures: one is related to the large-M expansion of joint cumulants of traces of powers of Q, which generalizes and implies a previous conjecture of Cunden, Mezzadri, Vivo and Simm; the other concerns the tail of the distribution of traces of powers of Q.