Due to the excellent optoelectronic performance, perovskite photodetectors (PDs) have attracted a lot of attention in recent years. However, the response speed of such a device is still relatively low. To overcome this defect, the intrinsic mechanisms within the perovskite PDs have to be uncovered. Here a comprehensive optoelectronic simulation is implemented to examine the transient response of the perovskite PDs, where the dynamics of both carriers (electrons/hoes) and ions (anions/cations) are taken into account. The unmatched temporal responses between the photogenerated electrons and holes are found to severely limit the overall response speed. Unfortunately, solely manipulating the perovskite parameters (e.g., thickness, doping concentration, and ion migration) cannot effectively suppress such a detrimental effect. Further study indicates that the mismatch in temporal response originates from the different electrical properties of the opposite charge transport layers, and can only be alleviated by balancing the corresponding carrier extraction capabilities. A roadmap for improving the response speed of the perovskite PDs is finally proposed, which shows the greatly shortened response time from 118.21 to 7.45 ns.