Artificial Intelligence of Things (AIoT) Enabled Virtual Shop Applications Using Self‐Powered Sensor Enhanced Soft Robotic Manipulator

Abstract

Rapid advancements of artificial intelligence of things (AIoT) technology pave the way for developing a digital-twin-based remote interactive system for advanced robotic-enabled industrial automation and virtual shopping. The embedded multifunctional perception system is urged for better interaction and user experience. To realize such a system, a smart soft robotic manipulator is presented that consists of a triboelectric nanogenerator tactile (T-TENG) and length (L-TENG) sensor, as well as a poly(vinylidene fluoride) (PVDF) pyroelectric temperature sensor. With the aid of machine learning (ML) for data processing, the fusion of the T-TENG and L-TENG sensors can realize the automatic recognition of the grasped objects with the accuracy of 97.143% for 28 different shapes of objects, while the temperature distribution can also be obtained through the pyroelectric sensor. By leveraging the IoT and artificial intelligence (AI) analytics, a digital-twin-based virtual shop is successfully implemented to provide the users with real-time feedback about the details of the product. In general, by offering a more immersive experience in human-machine interactions, the proposed remote interactive system shows the great potential of being the advanced human-machine interface for the applications of the unmanned working space.

Recent Progress in the Energy Harvesting Technology—From Self-Powered Sensors to Self-Sustained IoT, and New Applications

“…Then a digital twin unmanned warehouse system in virtual space is demonstrated, in which the objects gripped by the soft gripper in reality are recognized by the pre-trained support vector machine model and trigger the virtual objects gripped and placed in the corresponding boxes. Moreover, as shown in Figure 7c, Sun et al proposed an artificial intelligence of things (AIoT) enabled virtual shop applications using self-powered sensors enhanced soft robotic manipulator [259]. The patterned-electrode tactile TENG (T-TENG) sensors and the length TENG sensors are equipped to sense motions of the gripper, and 15 channels' grasping data are collected and trained in the 1D-CNN ML algorithm for automatic feature extraction, and calculated accuracy for six spherical objects and three oval objects can reach up to 96.1%.…”

“…(c) Artificial Intelligence of Things (AIoT) enabled virtual shop applications using self-powered sensor enhanced soft robotic manipulator (Reprinted with permission from ref. [259]. Copyright 2021 John Wiley & Sons).…”

Progress in the Triboelectric Human–Machine Interfaces (HMIs)-Moving from Smart Gloves to AI/Haptic Enabled HMI in the 5G/IoT Era

“…For realizing more advanced interactive functions, e.g., pose estimation, surface roughness perception and object recognition, AI-enabled analytics are needed, to capture more valuable features and endow the self-discrimination ability [225]. A TENG-enhanced smart soft robotic manipulator for AIoT enabled virtual shop applications was reported by Sun et al as shown in Figure 6d [274]. A TENG bending sensor consisting of a rotating gear was utilized to monitor the real-time deformation of the pneumatic finger according to the generated peak numbers during the stretching process, and a TENG tactile sensor with distributed electrodes design was used for contact position and area detection.…”

Recent Advances in Self-Powered Piezoelectric and Triboelectric Sensors: From Material and Structure Design to Frontier Applications of Artificial Intelligence

“…Digital twins represent an emerging concept. Sun et al [163] developed an intelligent soft robot manipulator. Combined with the Internet of things and artificial intelligence (AI), the manipulator can realize a virtual store based on digital twins and provide users with real-time feedback of product details.…”