A novel flame-retardant hsaltryptamine phosphaphenanthrene (HTD) was successfully synthesized and characterized by Fourier transform infrared spectrometry and nuclear magnetic resonance. HTD was used to improve flame retardancy of epoxy resin (EP). When the addition content of HTD reached 4 wt%, the EP/HTD thermoset passed a UL-94 V-1 rating and had limited oxygen index value of 29.2%. Compared to neat EP, the release of heat and smoke decreased significantly due to the gaseous-phase and condensed-phase flame-retardant mechanism of HTD. The char residue after cone calorimeter test was analyzed and the results showed that the char residue of EP/8%HTD thermoset exhibited homogeneously crosslinked char layer, which acted as a good barrier for the release of heat and oxygen. The transmittance and mechanical properties of EP/HTD thermosets had been tested. EP/HTD thermoset possessed high transparency and good mechanical properties. In comparison to neat EP, tensile strength, flexural strength, and Izod unnotched impact strength of EP with 8% loading of HTD was improved by 21.9%, 20.5%, and 17.4%, respectively.