Segmentation and tracking of vegetable plants by exploiting vegetable shape feature for precision spray of agricultural robots

Abstract

For robotic precision spray application in vegetable farms, simultaneous accurate instance segmentation and robust tracking of plants are of great importance and a prerequisite for the following spray action. With onboard cameras, agricultural robots can apply Multiple Object Tracking and Segmentation (MOTS) methods, for instance, segmentation and tracking of plants. By assigning a unique identification for each vegetable, it ensures the robot to spray each vegetable exactly once, while traversing along the farm rows. Conventional MOTS methods, which are mostly designed for tracking pedestrians or vehicles, usually extract their color and texture features for associating different targets in consecutive images. However, vegetable plants of the same species normally show similar color and texture, which leads to degraded performance when conventional MOTS methods are used. To solve the challenging problem of associating vegetables with similar color and texture in consecutive images, in this paper, a novel MOTS method that exploits contour and blob features is proposed, for instance, segmentation and tracking of multiple vegetable plants. The method takes advantage of the fact that different plants normally possess different shape contours and blob properties. With images captured on top of them, these features of the same plant show little difference in consecutively captured images. Comprehensive experiments including ablation studies are conducted, which prove its superior performance over two state‐of‐the‐art MOTS methods. Compared with the conventional MOTS methods, the proposed method is able to re‐identify objects which have gone out of the camera field of view and re‐appear again using the proposed data association strategy, which is important to ensure each vegetable be sprayed only once when the robot travels back and forth. Although the method is tested on lettuce farm, it can be applied to other similar vegetables, such as broccoli and canola. Both the code and the dataset of this paper are publicly released for the benefit of the community: https://github.com/NanH5837/LettuceMOTS.