The whole-cell scale spatial organization of lysosomes is regulated by their bidirectional motility on microtubule tracks. Small GTP-binding (G) protein, Arl8b, stimulates the anterograde transport of lysosomes by recruiting adaptor protein SKIP (also known as PLEKHM2), which in turn couples the microtubule motor kinesin-1. Here, we have identified an Arl8b effector, RUN and FYVE domain-containing protein family member 3, RUFY3, which drives the retrograde transport of lysosomes. Artificial targeting of RUFY3 to the surface of mitochondria was sufficient to drive their perinuclear positioning. We find that RUFY3 interacts with the JIP4-Dynein-Dynactin complex and mediates Arl8b association with the retrograde motor complex. The mobile fraction of the total lysosomes per cell was significantly enhanced upon RUFY3 depletion, suggesting that RUFY3 maintains the lysosomes clustering within the perinuclear cloud. Expectedly, RUFY3 knockdown disrupted the perinuclear positioning of lysosomes upon nutrient starvation and/or serum depletion, although lysosome continued to undergo fusion with autophagosomes. Interestingly, lysosome fission events were more frequent in RUFY3-depleted cells and accordingly, there was a striking reduction in lysosome size, an effect that was also observed in dynein and JIP4 depleted cells. These findings indicate that the dynein-dependent “perinuclear cloud” arrangement of lysosomes also regulates the size of these proteolytic compartments and, likely, their cellular roles.