New semiconductor materials are envisaged in numerous high-performance applications for which the expected device or circuit performances cannot be achieved with silicon. In this context of growing use of new and specific semiconductors, the question of their susceptibility to natural radiation, primarily to atmospheric neutrons, is posed for high-reliability-level application domains. This numerical simulation work precisely examines nuclear events resulting from the interaction of atmospheric neutrons at the terrestrial level with a target layer composed of various group-IV and III-V semiconductor materials including silicon, germanium, silicon carbide, carbon-diamond, gallium arsenide, and gallium nitride materials. Using extensive Geant4 simulations and in-depth data analysis, this study provides an accurate and fine comparison between the neutron interaction responses of these different semiconductors in terms of nuclear processes, recoil products, secondary ion production, and fragment energy distributions. Implications of these results on the rate of single-event transient effects at the device or circuit level are also discussed.