In order to solve the problem of time-consuming, labor-intensive and inefficient traditional packaging and sorting operations, an intelligent sorting system is designed based on machine vision theory and manipulator control technology. This paper introduces the design and implementation strategy of intelligent material sorting system in detail, and expounds the design and implementation of visual processing based on Res Net18 neural network and mechanical arm loading, communication, identification and sorting procedures under the framework of Python. The effectiveness and accuracy of the system are verified based on specific experiments. In the sorting test of four kinds of stacking materials, the recognition time is within 100ms, and the recognition accuracy rate is more than 99%, which meets the system design requirements. During the whole process, the intelligent material sorting system operates smoothly without problems such as holding brake and material dropping, and the sorting and grasping accuracy is high, which meets the requirements of high accuracy and high stability in the sorting field.