New and efficient sensors of nerve agents are urgently demanded to prevent them from causing mass casualties in war or terrorist attacks. So, in this work, a novel hierarchical nanoheterostructure was synthesized via the direct growth of α-Fe2O3 nanorods onto multiwall carbon nanotube (MWCNT) backbones. Then, the composites were functionalized with hexafluoroisopropanol (HFIP) and successfully applied to detect dimethyl methylphosphonate (DMMP)-sarin simulant gas. The observations show that the HFIP-α-Fe2O3@MWCNT hybrids exhibit outstanding DMMP-sensing performance, including low operating temperature (220 °C), high response (6.0 to 0.1 ppm DMMP), short response/recovery time (8.7 s/11.9 s), as well as low detection limit (63.92 ppb). The analysis of the sensing mechanism demonstrates that the perfect sensing performance is mainly due to the synergistic effect of the chemical interaction of DMMP with the heterostructure and the physical adsorption of DMMP by hydrogen bonds with HFIP that are grafted on the α-Fe2O3@MWCNTs composite. The huge specific surface area of HFIP-α-Fe2O3@MWCNTs composite is also one of the reasons for this enhanced performance. This work not only offers a promising and effective method for synthesizing sensitive materials for high-performance gas sensors but also provides insight into the sensing mechanism of DMMP.