Consider two random vectors r x P R p and r y P R q of the forms rx " Az `C1{2 1 x and r y " Bz `C1{2 2 y, where x P R p , y P R q and z P R r are independent random vectors with i.i.d. entries of zero mean and unit variance, C1 and C2 are p ˆp and q ˆq deterministic population covariance matrices, and A and B are p ˆr and q ˆr deterministic factor loading matrices. With n independent observations of pr x, r yq, we study the sample canonical correlations between rx and r y. We consider the high-dimensional setting with finite rank correlations, that is, p{n Ñ c1 and q{n Ñ c2 as n Ñ 8 for some constants c1 P p0, 1q and c2 P p0, 1 ´c1q, and r is a fixed integer. Let t1 ě t2 ě ¨¨¨ě tr ě 0 be the squares of the nontrivial population canonical correlation coefficients between rx and r y, and let r λ1 ě r λ2 ě ¨¨¨ě r λp^q ě 0 be the squares of the sample canonical correlation coefficients. If the entries of x, y and z are i.i.d. Gaussian, then the following dichotomy has been shown in [7] for a fixed threshold tc P p0, 1q: for any 1 ď i ď r, if ti ă tc, then r λi converges to the right-edge λ`of the limiting eigenvalue spectrum of the sample canonical correlation matrix, and moreover, n 2{3 p r λi ´λ`q converges weakly to the Tracy-Widom law; if ti ą tc, then r λi converges to a deterministic limit θi P pλ`, 1q that is determined by c1, c2 and ti. In this paper, we prove that these results hold universally under the sharp fourth moment conditions on the entries of x, y and z. Moreover, we prove the results in full generality, in the sense that they also hold for near-degenerate ti's and for ti's that are close to the threshold tc. Finally, we also provide almost sharp convergence rates for the sample canonical correlation coefficients under a general a-th moment assumption.