Plastic has been detected in the ocean in most locations where scientists have looked for it. While ubiquitous in the environment, plastic pollution is heterogeneous, and plastics of varying composition, shape, and size accumulate differently in the global ocean. Many physical and biological processes influence the transport of plastics in the marine environment. Here we focus on physical processes and how they can naturally sort floating plastics at the ocean surface and within its interior. We introduce a new open-source GPU-accelerated numerical model, ADVECT, which simulates the three-dimensional dispersal of large arrays of modelled ocean plastics with varying size, shape, and density. We use this model to run a global simulation and find that buoyant particles are sorted in the ocean according to their size, both at the surface due to wind-driven drift and in the water column due to their rising velocity. Finally, we compare our findings with recent literature reporting the size distribution of plastics in the ocean and discuss which observations can and cannot be explained by the physical processes encoded in our model.