Differentiable Rendering: A Survey

Kato, Hiroharu; Beker, Deniz; Morariu, Mihai; Ando, Takahiro; Matsuoka, Toru; Kehl, Wadim; Gaidon, Adrien

doi:10.48550/arxiv.2006.12057

Cited by 24 publications

(28 citation statements)

References 112 publications

(273 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here R(S; θ i ) is the rendering operator that projects a shape S from 'shape space' to 'projection space' based on CT gantry position θ i . To enable gradient backpropagation and optimizing via standard gradient descent methods, the rendering operator R should be differentiable [58].…”

Section: B Optimization Via Differentiable Renderingmentioning

confidence: 99%

Neural Computed Tomography

Gupta¹,

Colvert²,

Contijoch³

2022

Preprint

View full text Add to dashboard Cite

Motion during acquisition of a set of projections can lead to significant motion artifacts in computed tomography reconstructions despite fast acquisition of individual views. In cases such as cardiac imaging, motion may be unavoidable and evaluating motion may be of clinical interest. Reconstructing images with reduced motion artifacts has typically been achieved by developing systems with faster gantry rotation or using algorithms which measure and/or estimate the displacements. However, these approaches have had limited success due to both physical constraints as well as the challenge of estimating/measuring non-rigid, temporally varying, and patient-specific motions. We propose a novel reconstruction framework, NeuralCT, to generate time-resolved images free from motion artifacts. Our approaches utilizes a neural implicit approach and does not require estimation or modeling of the underlying motion. Instead, boundaries are represented using a signed distance metric and neural implicit framework. We utilize 'analysis-by-synthesis' to identify a solution consistent with the acquired sinogram as well as spatial and temporal consistency constraints. We illustrate the utility of NeuralCT in three progressively more complex scenarios: translation of a small circle, heartbeat-like change in an ellipse's diameter, and complex topological deformation. Without hyperparameter tuning or change to the architecture, NeuralCT provides high quality image reconstruction for all three motions, as compared to filtered backprojection using mean-square-error and Dice metrics. Code and supplemental movies are available at https://kunalmgupta.github.io/projects/NeuralCT.html.

show abstract

Section: B Optimization Via Differentiable Renderingmentioning

confidence: 99%

Neural Computed Tomography

Gupta¹,

Colvert²,

Contijoch³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our approach uses a neural 3D shape representation and dynamic neural networks for image based synthesis. A full review of these ideas is outside the scope of this paper, and we refer interested readers to [20] for classical IBR and [21], [22] for neural rendering. We consider the most closely related works below.…”

Section: Related Workmentioning

confidence: 99%

Recursive-NeRF: An Efficient and Dynamically Growing NeRF

Yang¹,

Zhou²,

Peng³

et al. 2021

Preprint

View full text Add to dashboard Cite

View synthesis methods using implicit continuous shape representations learned from a set of images, such as the Neural Radiance Field (NeRF) method, have gained increasing attention due to their high quality imagery and scalability to high resolution. However, the heavy computation required by its volumetric approach prevents NeRF from being useful in practice; minutes are taken to render a single image of a few megapixels. Now, an image of a scene can be rendered in a level-of-detail manner, so we posit that a complicated region of the scene should be represented by a large neural network while a small neural network is capable of encoding a simple region, enabling a balance between efficiency and quality. Recursive-NeRF is our embodiment of this idea, providing an efficient and adaptive rendering and training approach for NeRF. The core of Recursive-NeRF learns uncertainties for query coordinates, representing the quality of the predicted color and volumetric intensity at each level. Only query coordinates with high uncertainties are forwarded to the next level to a bigger neural network with a more powerful representational capability. The final rendered image is a composition of results from neural networks of all levels. Our evaluation on three public datasets shows that Recursive-NeRF is more efficient than NeRF while providing state-of-the-art quality. The code will be available at https://github.com/Gword/Recursive-NeRF.

show abstract

“…Even if the Adversarial Machine Learning community exploited rendered views of a 3D object to craft real objects that fool a classifier [11], most of the efforts are oriented towards the case of datasets of images. However, in the last years a large number of novel rendering schemes were proposed, ranging from Neural Renderers [12], [13], to more generic Differentiable Renderers [14], that allow the user to compute gradients with respect to the parameters of the renderer, including meshes, textures, and others. Of course, these renderers make it easier to alter elements belonging to the 3D world with the purpose of optimizing a target objective function, thus opening for deeper investigations on how Adversarial Machine Learning can impact 3D Virtual Environments.…”

Section: Twnmh2mentioning

confidence: 99%

Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects

Meloni¹,

Tiezzi²,

Pasqualini³

et al. 2021

Preprint

View full text Add to dashboard Cite

In the last few years, the scientific community showed a remarkable and increasing interest towards 3D Virtual Environments, training and testing Machine Learning-based models in realistic virtual worlds. On one hand, these environments could also become a mean to study the weaknesses of Machine Learning algorithms, or to simulate training settings that allow Machine Learning models to gain robustness to 3D adversarial attacks. On the other hand, their growing popularity might also attract those that aim at creating adversarial conditions to invalidate the benchmarking process, especially in the case of public environments that allow the contribution from a large community of people. Most of the existing Adversarial Machine Learning approaches are focused on static images, and little work has been done in studying how to deal with 3D environments and how a 3D object should be altered to fool a classifier that observes it. In this paper, we study how to craft adversarial 3D objects by altering their textures, using a tool chain composed of easily accessible elements. We show that it is possible, and indeed simple, to create adversarial objects using off-the-shelf limited surrogate renderers that can compute gradients with respect to the parameters of the rendering process, and, to a certain extent, to transfer the attacks to more advanced 3D engines. We propose a saliency-based attack that intersects the two classes of renderers in order to focus the alteration to those texture elements that are estimated to be effective in the target engine, evaluating its impact in popular neural classifiers.

show abstract

Differentiable Rendering: A Survey

Cited by 24 publications

References 112 publications

Neural Computed Tomography

Neural Computed Tomography

Recursive-NeRF: An Efficient and Dynamically Growing NeRF

Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects

Contact Info

Product

Resources

About