Aims. We modeled the transit signatures in the Lyman-alpha (Lyα) line of a putative Earth-sized planet orbiting in the habitable zone (HZ) of the M dwarf GJ 436. We estimated the transit depth in the Lyα line for an exo-Earth with three types of atmospheres: a hydrogen-dominated atmosphere, a nitrogen-dominated atmosphere, and a nitrogen-dominated atmosphere with an amount of hydrogen equal to that of the Earth. For all types of atmospheres, we calculated the in-transit absorption they would produce in the stellar Lyα line. We applied it to the out-of-transit Lyα observations of GJ 436 obtained by the Hubble Space Telescope and compared the calculated in-transit absorption with observational uncertainties to determine if it would be detectable. To validate the model, we also used our method to simulate the deep absorption signature observed during the transit of GJ 436b and showed that our model is capable of reproducing the observations. Methods. We used a direct simulation Monte Carlo (DSMC) code to model the planetary exospheres. The code includes several species and traces neutral particles and ions. It includes several ionization mechanisms, such as charge exchange with the stellar wind, photo-and electron impact ionization, and allows to trace particles collisions. At the lower boundary of the DSMC model we assumed an atmosphere density, temperature, and velocity obtained with a hydrodynamic model for the lower atmosphere. Results. We showed that for a small rocky Earth-like planet orbiting in the HZ of GJ 436 only the hydrogen-dominated atmosphere is marginally detectable with the Space Telescope Imaging Spectrograph (STIS) on board the Hubble Space Telescope (HST). Neither a pure nitrogen atmosphere nor a nitrogen-dominated atmosphere with an Earth-like hydrogen concentration in the upper atmosphere are detectable. We also showed that the Lyα observations of GJ 436b can be reproduced reasonably well assuming a hydrogendominated atmosphere, both in the blue and red wings of the Lyα line, which indicates that warm Neptune-like planets are a suitable target for Lyα observations. Terrestrial planets, on the other hand, can be observed in the Lyα line if they orbit very nearby stars, or if several observational visits are available.