A wedge clutch with unique features of self-reinforcement and small actuation force was designed. Its self-reinforcement feature, associated with different factors such as the wedge angle and friction coefficient, brings different dynamics and unstable problem with improper parameters. To analyze this system, a complete mathematical model of the actuation system is built, which includes the DC motor, the wedge mechanism, and the actuated clutch pack. By considering several nonlinear factors, such as the slip-stick friction and the contact or not of the clutch plates, the system is piecewise linear. Through the stability analysis of the linearized system in clutch slipping phase, the stable condition of the designed parameters is obtained asα>arctan(μc). The mathematical model of the actuation system is validated by prototype testing. And with the validated model, the system dynamics in both stable and unstable conditions is investigated and discussed in engineering side.