We establish a formula of the large N factorization of the modular S-matrix for the coupled representations in U (N ) Chern-Simons theory. The formula was proposed by Aganagic, Neitzke and Vafa, based on computations involving the conifold transition. We present a more rigorous proof that relies on the universal character for rational representations and an expression of the modular S-matrix in terms of the specialization of characters.