We conduct a quantitative analysis of the star formation history (SFH) of the Milky Way's bulge by exploiting the constraining power of its stellar [Fe/H] and [Mg/Fe] distribution functions. Using APOGEE data, we confirm the previously-established bimodal [Mg/Fe]-[Fe/H] distribution within 3 kpc of the inner Galaxy. Compared to that in the solar vicinity, the high-α population in the bulge peaks at a lower [Fe/H]. To fit these observations, we use a simple but flexible star formation framework, which assumes two distinct stages of gas accretion and star formation, and systematically evaluate a wide multi-dimensional parameter space. We find that the data favor a three-phase SFH that consists of an initial starburst, followed by a rapid star formation quenching episode and a lengthy, quiescent secular evolution phase. The metal-poor, high-α bulge stars ([Fe/H]<0.0 and [Mg/Fe]>0.15) are formed rapidly (< 2 Gyr) during the early starburst. The density gap between the high-and low-α sequences is due to the quenching process. The metal-rich, low-α population ([Fe/H]>0.0 and [Mg/Fe]<0.15) then accumulates gradually through inefficient star formation during the secular phase. This is qualitatively consistent with the early SFH of the inner disk. Given this scenario, a notable fraction of young stars (age < 5 Gyr) is expected to persist in the bulge. Combined with extragalactic observations, these results suggest that a rapid star formation quenching process is responsible for bimodal distributions in both the MWs stellar populations and in the general galaxy population and thus plays a critical role in galaxy evolution.