Aimed at solving the problems of frequent handover and large overheads for positioning in 6G, this paper proposes a reinforcement learning (RL)-based visible light positioning (VLP) handover scheme by means of a stereo camera system in which a lower handover rate and higher positioning accuracy were achieved simultaneously. Because of the randomness of the distribution location of indoor light sources and obstacles, even if the parameters of light sources and receivers are consistent, users cannot rely on unitary and invariable parameters to determine whether to change the access point (AP) in the process of moving, especially in some special locations. The proposed scheme, which decomposed the user's moving track at different speeds in a VLP system to optimize selecting the AP by using RL at each step, was exhibited and tested. Experimental results show that the proposed scheme achieved millimeter-level positioning accuracy, improved the normalized reward by over 40% and reduced the handover rate by 87% and 78% compared to the immediate handover (IHO) and dwell handover (DHO) methods concurrently.