Purpose This paper aims to present the impact analysis of payload rendezvous with tethered satellite system and the design of an adaptive sliding mode controller which can deal with mass parameter uncertainty of targeted payload, so that the proposed cislunar transportation scheme with spinning tether system could be extended to a wider and more practical range. Design/methodology/approach In this work, dynamical model is first derived based on Langrangian equations to describe the motion of a spinning tether system in an arbitrary Keplerian orbit, which takes the mass of spacecraft, tether and payload into account. Orbital design and optimal open-loop control for the payload tossed by the spinning tether system are then presented. The real payload rendezvous impact around docking point is also analyzed. Based on reference acceleration trajectory given by optimal theories, a sliding mode controller with saturation functions is designed in the close-loop control of payload tossing stage under initial disturbance caused by actual rendezvous error. To alleviate the influence of inaccurate/unknown payload mass parameters, the adaptive law is designed and integrated into sliding mode controller. Finally, the performance of the proposed controller is evaluated using simulations. Simulation results validate that proposed controller is found effective in driving the spinning tether system to carry payload into desired cislunar transfer orbit and in dealing with payload mass parameter uncertainty in a relatively large range. Findings The results show that unideal rendezvous manoeuvres have significant impact on in-plane motion of spinning tether system, and the proposed adaptive sliding mode controller with saturation functions not only guarantees the stability but also provides good performance and robustness against the parameter and unstructured uncertainties. Originality/value This work addresses the analysis of actual impact on spinning tether system motion when payload is docking with system within tolerated docking window, rather than at the particular ideal docking point, and the robust tracking control of deep-space payload tossing missions with the spinning tether system using the adaptive sliding mode controller dealing with parameter uncertainties. This combination has not been proposed before for tracking control of multivariable spinning tether systems.