Due to the extraordinary chemical and physical properties, ternary chalcogenides with wide‐range bandgap, unique structures, and exceptional photoelectric properties have attracted great interest recently. Among them, Chromium thiophosphate (CrPS4) is a promising semiconductor with excellent optical, mechanical, and electronic properties and has a lower lattice symmetry resulting in significant in‐plane anisotropy. Here, the polarization‐dependent nonlinear optical (NLO) responses of CrPS4 flake are investigated. Based on polarization‐resolved Raman spectroscopy, which identifies the crystal axis of sample, the anisotropic linear absorption spectra, I‐scan technique, and pump‐probe measurements are utilized to investigate the optical absorption information of CrPS4. The polarization‐dependent NLO responses are found originating from the anisotropic optical transition probability. And the polar optical phonons play an important role in carrier relaxation processes. Furthermore, CrPS4‐based all‐optical switches are proposed, which can modulate the transmittance of signal light (CW laser) by changing the polarization of switch light (fs laser). This work provides insights into polarization‐dependent CrPS4‐based devices, which is important to promote the potential applications in optical modulator, versatile optoelectronic detection, and so on.