At present, the sorting of agricultural products in China mainly relies on manual labour, which results in low efficiency, and the development of corresponding automatic equipment lags behind. The grasping method based on machine vision has been widely used in industry, and can provide a reference for the automatic sorting of agricultural products. In this paper, an automatic sorting device for agricultural products was designed. The grasping mechanism adopted a 4-degree-of-freedom (4-DOF) manipulator, and the machine vision control system adopted a monocular camera, which can realize the positioning and classification of the grasp-target. First, the geometric model of the manipulator was established, and the kinematics model of the manipulator was established via the Denavit-Hartenberg (D-H) parameter method. Next, the kinematics analysis and verification were carried out. Then, Zhang Zhengyou calibration method was used to calibrate the camera. An image processing method based on histogram correction was proposed. Based on this, a target positioning algorithm based on the pinhole imaging principle and a target classification algorithm based on the area threshold were established. Finally, an automatic sorting test platform for agricultural products using a visual servo was built. Target classification, positioning and sorting tests were conducted using tomatoes and oranges as the test objects. The test results show that the success rate of the target positioning is close to 98%, that of the target classification is close to 98% and that of the grasping is close to 95%. Furthermore, the sorting time of a single target object can be as fast as 1 second, which can meet the requirements of automatic sorting for common agricultural products. The automatic sorting device for agricultural products has a simple structure, reliable performance and low costs. The structure and algorithms proposed in this paper are simple, reliable, and highly efficient and thus can easily realize technology transplantation. These relevant methods provide a theoretical reference for the development of an automatic sorting device for agricultural products.