We compute the exact effective action for N = 3 U(N ) k and N = 4, 6 U(N ) k × U(N ′ ) −k Chern-Simons theories with minimal matter content in the 't Hooft vector model limit under which N and k go to infinity holding N/k, N ′ fixed. We also extend this calculation to N = 4, 6 mass deformed case. We show that those large N effective actions except mass-deformed N = 6 case precisely reduce to that of N = 2 U(N ) k Chern-Simons theory with one fundamental chiral field up to overall multiple factor. By using this result we argue the thermal free energy and self-duality of the N = 3, 4, 6 Chern-Simons theories including the N = 4 mass term reduce to those of the N = 2 case under the limit.