Tool-Use Model to Reproduce the Goal Situations Considering Relationship Among Tools, Objects, Actions and Effects Using Multimodal Deep Neural Networks

Abstract

We propose a tool-use model that enables a robot to act toward a provided goal. It is important to consider features of the four factors; tools, objects actions, and effects at the same time because they are related to each other and one factor can influence the others. The tool-use model is constructed with deep neural networks (DNNs) using multimodal sensorimotor data; image, force, and joint angle information. To allow the robot to learn tool-use, we collect training data by controlling the robot to perform various object operations using several tools with multiple actions that leads different effects. Then the tool-use model is thereby trained and learns sensorimotor coordination and acquires relationships among tools, objects, actions and effects in its latent space. We can give the robot a task goal by providing an image showing the target placement and orientation of the object. Using the goal image with the tool-use model, the robot detects the features of tools and objects, and determines how to act to reproduce the target effects automatically. Then the robot generates actions adjusting to the real time situations even though the tools and objects are unknown and more complicated than trained ones.

Potential of digital chest radiography-based deep learning in screening and diagnosing pneumoconiosis: An observational study

“…This improves network reliability and accurately represents the input image, addressing the problem of the "black box" principle. [28] In this study, we developed an efficient deep learning method that eliminates the need for manual feature recognition, reducing the workload of diagnosing pneumoconiosis (the outline of the study design is given in Fig. 1).…”

“…This improves network reliability and accurately represents the input image, addressing the problem of the “black box” principle. [ 28 ]…”

Potential of digital chest radiography-based deep learning in screening and diagnosing pneumoconiosis

“…The deep learning network is an advanced technique for studying the aforementioned features and can offer heat maps that visualize the interpretation of model decisions. Thus, an improvement of the network reliability and an accurate representation of the content of the input image data solves the problem of the "black box" principle [20]. By comparing the performance of three convolutional neural networks (CNNs), we aimed to establish an end-to-end automatic pneumoconiosis medical image classi cation model based on deep learning and evaluate the feasibility of the model for pneumoconiosis classi cation.…”