Context. This work uses the measured luminosity vs. displacement (L X vs. R) distribution of high-mass X-ray binaries (HMXBs) to constrain the dispersion of kick velocity σ kick , which is an important parameter affecting the system velocity of a binary, and hence its spatial offset from the point of origin. Aims. The aim is to constrain the natal kicks and discriminate between models by comparing the observed L X vs. R distributions with the theoretical simulations. Methods. Using an up-to-date evolutionary population synthesis technique, the spatial offsets of HMXBs are modeled for a range of theoretical models describing the natal kicks, including different choices of the dispersion of kick velocity σ kick , as well as different theoretical treatments for black hole (BH) natal kicks. Results. The study shows that the value of σ kick for neutron stars (NSs) is constrained to be greater than ∼100 km s −1 , while σ kick on the order of several tens of km s −1 may be excluded, though a low or absent natal kick for electron capture supernovae NSs is permitted. In particular, BH natal kicks are found not indispensable to account for the L X vs. R distributions. It is more interesting that full BH natal kicks (i.e., similar to those that NSs may receive) are likely to be ruled out in this study, which is in contrast with the recent finding to explain the observed distribution of low-mass X-ray binaries hosting BHs.