In this paper the potentiality of impact avalanche transit time (IMPATT) devices based on different semiconductor materials such as GaAs, Si, InP, 4H-SiC and Wurtzite-GaN (Wz-GaN) has been explored for operation at terahertz frequencies. Drift-diffusion model is used to design double-drift region (DDR) IMPATTs based on different materials at millimeter-wave (mm-wave) and terahertz (THz) frequencies. The performance limitations of these devices are studied from the avalanche response times at different mm-wave and THz frequencies. Results show that the upper cut-off frequency limits of GaAs and Si DDR IMPATTs are 220 GHz and 0.5 THz, respectively, whereas the same for InP and 4H-SiC DDR IMPATTs is 1.0 THz. Wz-GaN DDR IMPATTs are found to be excellent candidate for generation of RF power at THz frequencies of the order of 5.0 THz with appreciable DC to RF conversion efficiency. Further, it is observed that up to 1.0 THz, 4H-SiC DDR IMPATTs excel Wz-GaN DDR IMPATTs as regards their RF power outputs. Thus, the wide bandgap semiconductors such as Wz-GaN and 4H-SiC are highly suitable materials for DDR IMPATTs at both mm-wave and THz frequency ranges.