Position-based visual servoing for pallet picking by an articulated-frame-steering hydraulic mobile machine

Abstract

This paper addresses a visual servoing problem for a mobile manipulator. Specifically, it investigates pallet picking by using visual feedback using a fork lift truck. A manipulator with limited degrees of freedom and differential constraint mobility together with large dimensions of the machine require reliable visual feedback (pallet pose) from relatively large distances. To address this challenge, we propose a control architecture composed of three main sub-systems: (1) pose estimation: body and fork pose estimation in the pallet frame; (2) path planning: from the current pose to the origin (pallet frame); and (3) feedback motion control. In this architecture, the pallet becomes the local earth fixed frame in which poses are resolved and plans are formulated. Choosing the pallet as the origin provides a natural framework for fusing the wheel odometry/inertial sensor data with vision, and planning is required only once the pallet is detected for the first time (because the target is always the origin). Visual pallet detection is non-real-time and unreliable, especially owing to large distances, unfavorable vibrations, and fast steering. To address these issues, we introduce a simple and efficient method that integrates the vision output with odometry and realizes smooth and non-stop transition from global navigation to visual servoing. Real-world implementation on a small-sized forklift truck demonstrates the efficacy of the proposed visual servoing architecture.

Detection, localisation and tracking of pallets using machine learning techniques and 2D range data

“…Eventually, these features are then fed to a network with two main tasks, namely regression and classification. The regression output determines the predicted bounding boxes, each with a form of [x min , y min , x len , y len ], while the output of the classification network is the value o indicating whether each predicted bounding box contains an object, according to (1). This implies that Faster R-CNNs achieve efficient and fully end-to-end training, as a single CNN is used for region proposal and classification.…”

“…The training process runs for 20 epochs, leading to an approximately 45-minute training time on our workstation. Once training is complete and all the ROIs are generated, the corresponding bounding boxes are additionally filtered using non-maximum suppression with an overlap threshold of 0.3, as shown in (1). Figure 4(a) shows a sample image, the ROIs detected in it, and their corresponding confidence scores, while Figure 4(b) shows only the ROIs remaining after suppression.…”

Fault Tolerant Control Architecture Design for Mobile Manipulation in Scientific Facilities

“…Manipulators are usually provided together with their specialized controllers. However, coordinated tasks have to be performed synchronously between the path follower for the mobile base and the arm controller [27]. We use a mobile manipulator controller to close the outer control loop that follows the task commands distributed among the mobile base and the arm in a similar manner to [28].…”

A safe autonomous stacker in human shared workspaces

“…The solution to this problem typically relies on the use of laser scanners [7], 2D cameras [21], 3D cameras [20], a combination of the former [12], or even using infrared systems [17]. The pick phase is generally translated to a servo command problem, with visual servoing being a typical choice [1], [15]. Others still rely on pallet markers to navigate during the picking phase [7].…”