As the second-generation compound semiconductor material, indium phosphide (InP) has strong irradiation resistance and high photoelectric conversion efficiency. It has advantages in the field of photonics and radio frequency. In atmospheric space, high-energy cosmic rays enter into the earth's atmosphere and interact with nitrogen (N), oxygen (O) and other elements to produce secondary cosmic rays. The irradiation particles in the atmosphere are mainly neutrons because the penetration of charged particles is weak. The InP semiconductor devices are affected by atmospheric neutron irradiation of various energy from all directions, which results in the internal defects in InP crystals, the degradation of device performance and the reduction of device lifetime. In this paper, Monte Carlo simulation software Geant4 is used to simulate the neutron irradiation effect, and the initial state distribution of displacement damage caused by neutrons with different energy is obtained, including the distribution of non-ionized energy loss (NIEL) with depth, the relationship between NIEL and the energy of incident neutrons, and the type, number and energy of primary knock-on atoms (PKA). The results show that (1) the NIEL is uniformly distributed when material thickness is on the order of μm and for the material thickness on the order of cm and more, the NIEL decreases as the depth increases and can be reduced to zero when the target material is thick enough; (2) by analyzing the NIEL produced by 1–20 MeV neutrons incident on 3-μm InP and their distribution with depth, it is found that the NIEL first increases and then decreases with incident neutron energy increasing. This trend is caused mainly by PKA produced through the inelastic scattering reaction; (3) by analyzing the type and the energy of PKA produced by 1–20 MeV neutrons incident on 3 μm InP, it is found that the PKA of In/P accounts for a large proportion, which causes displacement damage mainly, and the higher the neutron energy, the richer the variety of PKA is and the greater the maximum kinetic energy of PKA, but the PKAs mainly distribute in the low energy part. The present research has theoretical and guiding value for the long-term application of InP-based 5G devices in atmospheric neutron irradiation environment.