Sorting granular materials such as ores, coffee beans, cereals, gravels and pills is essential forapplications in mineral processing, agriculture and waste recycling. Existing sorting methods are based on the detection of contrast in grain properties including size, colour, density and chemical composition. However, many grain properties cannot be directly detected in-situ, which significantly impairs sorting efficacy. We show here that a simple neural network can infer contrast in a wide range of grain properties by detecting patterns in their observable kinematics. These properties include grain size, density, stiffness, friction, dissipation and adhesion. This method of classification based on behaviour can significantly widen the range of granular materials that can be sorted. It can similarly be applied to enhance the sorting of other particulate materials including cells and droplets in microfluidic devices.