Weeds have a detrimental effect on crop yield. However, the prevailing chemical weed control methods cause pollution of the ecosystem and land. Therefore, it has become a trend to reduce dependence on herbicides; realize a sustainable, intelligent weed control method; and protect the land. In order to realize intelligent weeding, efficient and accurate crop and weed recognition is necessary. Convolutional neural networks (CNNs) are widely applied for weed and crop recognition due to their high speed and efficiency. In this paper, a multi-path input skip-residual network (SkipResNet) was put forward to upgrade the classification function of weeds and crops. It improved the residual block in the ResNet model and combined three different path selection algorithms. Experiments showed that on the plant seedling dataset, our proposed network achieved an accuracy of 95.07%, which is 0.73%, 0.37%, and 4.75% better than that of ResNet18, VGG19, and MobileNetV2, respectively. The validation results on the weed–corn dataset also showed that the algorithm can provide more accurate identification of weeds and crops, thereby reducing land contamination during the weeding process. In addition, the algorithm is generalizable and can be used in image classification in agriculture and other fields.