We report on polarized Raman scattering on orthorhombic RCrO3 (R = La, Pr, Nd, and Sm) single crystals at room temperature. The analysis of polarized Raman spectra as a function of polarizer and analyzer angles with respect to crystallographic axes allowed to observe in these flux grown crystals phonon modes that were not detected yet. On the basis of both Raman selection rules and lattice dynamics calculations within density functional perturbations theory framework, the observed Ag, B1g, B2g, and B3g phonon modes were unambiguously assigned. Phonon mode wave number evolutions as a function of orthorhombic distortion, related to rare earth ionic radius, were investigated and highlighted a complex behavior for several low wave number modes allowed crossing. For the modes of same symmetry, a mode mixing behavior is observed, and numerical calculations based on two or three coupling systems were performed to fit experimental data and to extract the coupling constants.