We use a Monte-Carlo model to simulate semi-classical photo-carrier dynamics of InAs, InGaAs, and GaAs that leads to terahertz emission. We compare the emission power of all three semiconductors as a function of excitation photon energy finding that the carrier excess excitation energy is more relevant to explain their performance difference than their mobilities. We conclude that ballistic transport after photoexcitation is the dominant mechanism for terahertz emission instead of diffusion driven or surface field driven charge separation, which were traditionally considered the most relevant mechanisms.