Recent collapses of Session Initiation Protocol (SIP) servers indicate that the built-in SIP overload control mechanism cannot mitigate overload effectively. In this paper, we propose a new SIP overload control algorithm by introducing a novel analytical approach to model the dynamic behavior of a SIP network where each server has a finite buffer. Three key breakthroughs of our modeling approach are the formulations of the message loss process, message retransmission process, and the complex departure process through detailed analysis. Our modeling results indicate that retransmissions triggered by the queuing delay are redundant, thus we propose a feedback control mechanism that regulates the retransmission message rate to mitigate the overload. We then demonstrate how to extend our analytical approach to the modeling of our overload control solution. Simulation based on this analytical model runs much faster than eventdriven simulation, which needs to track thousands of retransmission timers for outstanding messages and may crash a simulator due to limited computation resources. Performance evaluation demonstrates that: (1) without the control algorithm applied, the overload at a downstream server may propagate to its upstream servers and cause widespread network failure; (2) in the case of short-term overload, our feedback control solution can mitigate the overload effectively without rejecting calls intentionally or reducing network utilization, thus avoiding the disadvantages of existing overload control solutions. In addition, compared with the pushback solution, our retransmission-based solution achieves a better trade-off between the speed to cancel the overload and the call rejection rate when an overload lasts a short period.