Predicting 3D shapes, masks, and properties of materials inside transparent containers, using the TransProteus CGI dataset

Eppel, Sagi; Xu, Haoping; Wang, Yi Ru; Aspuru‐Guzik, Alán

doi:10.1039/d1dd00014d

Cited by 10 publications

(9 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, 2D to 3D reconstruction technologies applicable to different application scenarios will be quite different. For example, in [ 2 ], a method is proposed to predict the liquid or solid in transparent vessels to XYZ maps. This method can be applied, for example, to the task of a robot arm taking containers and pouring liquid.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Single Stage and Single View 3D Point Cloud Reconstruction Network Based on DetNet

Zhu

2022

Sensors

View full text Add to dashboard Cite

It is a challenging problem to infer objects with reasonable shapes and appearance from a single picture. Existing research often pays more attention to the structure of the point cloud generation network, while ignoring the feature extraction of 2D images and reducing the loss in the process of feature propagation in the network. In this paper, a single-stage and single-view 3D point cloud reconstruction network, 3D-SSRecNet, is proposed. The proposed 3D-SSRecNet is a simple single-stage network composed of a 2D image feature extraction network and a point cloud prediction network. The single-stage network structure can reduce the loss of the extracted 2D image features. The 2D image feature extraction network takes DetNet as the backbone. DetNet can extract more details from 2D images. In order to generate point clouds with better shape and appearance, in the point cloud prediction network, the exponential linear unit (ELU) is used as the activation function, and the joint function of chamfer distance (CD) and Earth mover’s distance (EMD) is used as the loss function of 3DSSRecNet. In order to verify the effectiveness of 3D-SSRecNet, we conducted a series of experiments on ShapeNet and Pix3D datasets. The experimental results measured by CD and EMD have shown that 3D-SSRecNet outperforms the state-of-the-art reconstruction methods.

show abstract

Section: Related Workmentioning

confidence: 99%

“…In many tasks, such as virtual reality [ 1 ], experimental assistance [ 2 ], and robot navigation [ 3 ], a detailed 3D model is required, however, the facilities required to sample 3D models from the real world are costly. Moreover, it is uneconomical to manually reconstruct 3D models from 2D maps on a large scale.…”

Section: Introductionmentioning

confidence: 99%

A Single Stage and Single View 3D Point Cloud Reconstruction Network Based on DetNet

Zhu

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…As a result, the first stage of the cascaded model is a mask R-CNN [11]. Given that the mask R-CNN is currently the state-of-the-art method for instance segmentation with respect to a large array of image segmentation tasks [11], this work adopts the pre-trained model on the TransProteus dataset [7]. The role of this model is to find the region and boundaries of the glass laboratory vial in the image that is recorded directly from the end-effector of the Panda Emika Franka robotic arm.…”

Section: A Cascaded Modelmentioning

confidence: 99%

“…Recently, vision methods based on deep neural networks and convolutional neural networks (CNNs) have proven effective in laboratory and medical environments [6]. Such methods have gained traction for visual recognition of lab hardware [7] and have shown promising results in challenging real-world settings. This is remarkable since laboratory glassware and many solvents are in fact transparent and hence pose a significant challenge for classical machine vision algorithms.…”

Section: Introductionmentioning

confidence: 99%

SOLIS: Autonomous Solubility Screening using Deep Neural Networks

Pizzuto¹,

Berardinis²,

Longley³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accelerating material discovery has tremendous societal and industrial impact, particularly for pharmaceuticals and clean energy production. Many experimental instruments have some degree of automation, facilitating continuous running and higher throughput. However, it is common that sample preparation is still carried out manually. This can result in researchers spending a significant amount of their time on repetitive tasks, which introduces errors and can prohibit production of statistically relevant data. Crystallisation experiments are common in many chemical fields, both for purification and in polymorph screening experiments. The initial step often involves a solubility screen of the molecule; that is, understanding whether molecular compounds have dissolved in a particular solvent. This usually can be time consuming and work intensive. Moreover, accurate knowledge of the precise solubility limit of the molecule is often not required, and simply measuring a threshold of solubility in each solvent would be sufficient. To address this, we propose a novel cascaded deep model that is inspired by how a human chemist would visually assess a sample to determine whether the solid has completely dissolved in the solution. In this paper, we design, develop, and evaluate the first fully autonomous solubility screening framework, which leverages state-of-the-art methods for image segmentation and convolutional neural networks for image classification. To realise that, we first create a dataset comprising different molecules and solvents, which is collected in a real-world chemistry laboratory. We then evaluated our method on the data recorded through an eye-in-hand camera mounted on a seven degree-of-freedom robotic manipulator, and show that our model can achieve 99.13% test accuracy across various setups, while being simple and fast to train and, as a result, easily transferable to a robotic platform.

show abstract

“…With respect to formulation development and optimization, liquid handling systems can be applied anywhere from formulation screening (i.e., pipetting drug and/or excipient solutions) and in vitro cytotoxicity evaluation to the preparation of analytical samples [37,38]. While not necessarily new, as robotic arms and certain automated capabilities have been a part of analytical laboratories for years, such systems continue to increase the range of tasks that can be performed and pave the way for the integration of different technologies (e.g., computer vision to recognize material properties) [39,40].…”

Section: Laboratory Automation -Strengthening Data Productionmentioning

confidence: 99%

Re-envisioning the Design of Nanomedicines: Harnessing Automation and Artificial Intelligence

Zaslavsky

Bannigan

Allen

2022

Preprint

View full text Add to dashboard Cite

Introduction Interest in nanomedicines has surged in recent years due to the critical role they have played in the COVID-19 pandemic. Nanoformulations can turn promising therapeutic cargo into viable products through improvements in drug safety and efficacy profiles. However, the developmental pathway for such formulations is non-trivial and largely reliant on trial-and-error. Beyond the costly demands on time and resources, this traditional approach may stunt innovation. The emergence of automation, artificial intelligence (AI) and machine learning (ML) tools, which are currently underutilized in pharmaceutical formulation development, offers a promising direction for an improved path in the design of nanomedicines. Areas covered This article highlights the potential of harnessing experimental automation and AI/ML to drive innovation in nanomedicine development. The discussion centers on the current challenges in drug formulation research and development, and the major advantages afforded through the application of data-driven methods. Expert opinion The development of integrated workflows based on automated experiments and AI/ML may accelerate nanomedicine development. A crucial step in achieving this is the generation of high-quality, accessible datasets. Future efforts to make full use of these tools can ultimately contribute to the development of more innovative nanomedicines and improved clinical translation of formulations that rely on advanced drug delivery systems.

show abstract

Predicting 3D shapes, masks, and properties of materials inside transparent containers, using the TransProteus CGI dataset

Abstract: We present TransProteus, a dataset, and methods for predicting the 3D structure and properties of materials inside transparent vessels from a single image. Manipulating materials in containers is essential in...

Cited by 10 publications

References 34 publications

A Single Stage and Single View 3D Point Cloud Reconstruction Network Based on DetNet

A Single Stage and Single View 3D Point Cloud Reconstruction Network Based on DetNet

SOLIS: Autonomous Solubility Screening using Deep Neural Networks

Re-envisioning the Design of Nanomedicines: Harnessing Automation and Artificial Intelligence

Contact Info

Product

Resources

About