Abstract:The availability and performance of laser-based adaptive optics (AO) systems are strongly dependent on the power and quality of the laser beam before being projected to the sky. Frequent and time-consuming alignment procedures are usually required in the laser systems with free-space optics to optimize the beam. Despite these procedures, significant distortions of the laser beam have been observed during the first two years of operation of the Gemini South multi-conjugate adaptive optics system (GeMS). A beam shaping concept with two deformable mirrors is investigated in order to provide automated optimization of the laser quality for astronomical AO. This study aims at demonstrating the correction of quasi-static aberrations of the laser, in both amplitude and phase, testing a prototype of this two-deformable mirror concept on GeMS. The paper presents the results of the preparatory study before the experimental phase. An algorithm to control amplitude and phase correction, based on phase retrieval techniques, is presented with a novel unwrapping method. Its performance is assessed via numerical simulations, using aberrations measured at GeMS as reference. The results predict effective amplitude and phase correction of the laser distortions with about 120 actuators per mirror and a separation of 1.4 m between the mirrors. The spot size is estimated to be reduced by up to 15% thanks to the correction. In terms of AO noise level, this has the same benefit as increasing the photon flux by 40%.