We study an impact of asymmetric dark matter on properties of the neutron stars and their ability to reach the two solar masses limit, which allows us to present a new upper constraint on the mass of dark matter particle. Our analysis is based on the observational fact of existence of three pulsars reaching this limit and on the theoretically predicted reduction of the neutron star maximal mass caused by accumulation of dark matter in its interior. Using modern data on spatial distribution of baryon and dark matter in the Milky Way we argue that particles of dark matter can not be heavier than 5 GeV. We also demonstrate that light dark matter particles with masses below 0.2 GeV can create an extended halo around the neutron star leading not to decrease, but to increase of its visible gravitational mass. Furthermore, we predict that high precision measurements of the neutron stars maximal mass near the Galactic center, will put a stringent constraint on the mass of the dark matter particle. This last result is particularly important to prepare ongoing, and future radio and X-ray surveys.