Terahertz radiation can be generated due to the instability of terahertz(THz) plasma waves in field-effect transistors (FET). In this work, we discuss the instability of THz plasma waves in the channel of FET with spin and quantum effects under non-ideal boundary conditions. We obtain a linear dispersion relation by using hydrodynamic equation, Maxwell equation and spin equation. The influence of source capacitance, drain capacitance, spin effects, quantum effects and channel width on the instability of THz plasma waves under the non-ideal boundary conditions is investigated in great detail. The results of numerical simulation show that the terahertz plasma wave is unstable when the source capacitance is greater than the drain capacitance. This finding provides a new idea for finding efficient THz radiation sources and opens up a new mechanism for the development of THz technology.