We describe the ring structure of the rational cohomology of the Torelli groups of the manifolds $\#^g S^n \times S^n$ in a stable range, for $2n \geq 6$. Some of our results are also valid for $2n=2$, where they are closely related to unpublished results of Kawazumi and Morita.